JMIR Publications

ABSTRACT

Background:

An estimated 93% of adults in the United States access the Internet, with up to 80% of them looking for health information[1]. However, only 12% of US adults are assessed to be proficient in health literacy to meaningfully interpret health information and make informed healthcare decisions. With the vast amount of health information available in multi-media format on social media platforms such as YouTube and Facebook, there is an urgent need and a unique opportunity to design an automated approach to curate online health information using multiple criteria to meet the health literacy needs of a diverse population.

Objective:

We focus on the understandability of online audio-visual content of YouTube videos on health topics. We draw on the Patient Education Material Assessment Tool (PEMAT), a systematic approach for audio-visual educational materials assessment, to develop an automated method to assess the understandability of YouTube videos on diabetes from a patient education perspective.

Methods:

We develop a human-in-the-loop assessment that explicitly focuses on the human-algorithm interaction combining PEMAT-based patient education constructs that are mapped to features extracted from the videos, annotations of the videos by domain experts, and co-training methods from machine learning to assess the understandability of diabetes videos and classify them. We further examine the impact of understandability on several dimensions of viewer engagement with the videos.

Results:

We collected 9,873 YouTube videos on diabetes using the search keywords extracted from a patient forum and reviewed by a medical expert. Our machine learning methods achieved a weighted precision of 0.84, a weighted recall of 0.79, and an F1 score of 0.81 in classifying video understandability and can effectively identify patient educational videos that medical experts would like to recommend for patients. The results show that the co-training method significantly improved the video understandability classification performance with limited labeled videos and that understandability has a positive impact on the view count, like count, and comment count of the videos.

Conclusions:

We develop a human-in-loop, scalable, generalizable and multi-modal algorithmic solution to evaluate the understandability of healthcare information on social media platforms. Assessing the educational value of videos in domains ranging from healthcare to education still requires domain expertise to gauge the content. The evaluation results suggest that our method demonstrated an optimal combination of human expert involvement and algorithmic decision support. The results also show that video understandability in health educational videos is not only critical to engagement on the social media platform but also valuable to medical experts when recommending curated content for patient education. Our proposed solution can also provide healthcare organizations with actionable guidance in designing and creating patient educational videos for the plethora of health conditions for which adequate educational materials do not currently exist.