JMIR Publications

ABSTRACT

Background:

An estimated 93% of adults in the United States access the Internet, with up to 80% of them looking for health information. However, only 12% of US adults are assessed to be proficient in health literacy to meaningfully interpret health information and make informed healthcare decisions. With the vast amount of health information available in multi-media format on social media platforms such as YouTube and Facebook, there is an urgent need and a unique opportunity to design an automated approach to curate online health information using multiple criteria to meet the health literacy needs of a diverse population.

Objective:

In this study, we aim to develop an automated approach to assess the understandability of patient educational videos according to PEMAT guideline and to evaluate the impact of video understandability on viewer engagement, offering insights for content creators on how to improve the engagement of their educational videos on user generated content platforms.

Methods:

We develop a human-in-the-loop, augmented intelligence approach that explicitly focuses on the human-algorithm interaction combining PEMAT-based patient education constructs mapped to features extracted from the videos, annotations of the videos by domain experts, and co-training methods from machine learning to assess the understandability of diabetes videos and to classify them. We further examine the impact of understandability on several dimensions of viewer engagement with the videos.

Results:

We collected 9,873 YouTube videos on diabetes using search keywords extracted from a patient-oriented forum and reviewed by a medical expert. Our machine learning methods achieved a weighted precision of 0.84, a weighted recall of 0.79, and an F1 score of 0.81 in classifying video understandability and can effectively identify patient educational videos that medical experts would like to recommend for patients. Videos rated as highly understandable had an average increase in view count (ATE: 2.55, p < 2e-16), like count (ATE: 2.95, p < 2e-16), and comment count (ATE: 3.10, p < 2e-16) compared to less understandable videos. Additionally, in a user study, four medical experts recommended 72% of the top 10 videos ranked by understandability compared to 40% of the top 10 videos ranked by YouTube’s default algorithm.

Conclusions:

We develop a human-in-loop, scalable, generalizable algorithmic solution to evaluate the understandability of healthcare information on the YouTube social media platform. The evaluation results suggest that our method demonstrated an optimal combination of human expert involvement and algorithmic decision support. The results also show that video understandability in health educational videos is not only critical to engagement on the social media platform but also valuable to medical experts when recommending curated content for patient education. Our proposed solution can also provide healthcare organizations with actionable guidance in designing and creating patient educational videos for the plethora of health conditions for which adequate educational materials do not currently exist.