JMIR Publications

ABSTRACT

Background:

Having well-informed patients is crucial in enhancing patients’ satisfaction, quality of life and health outcomes. Which in turn, optimises healthcare utilisation. Traditional methods of delivering information, such as booklets and leaflets, are often ineffective and can overwhelm patients. Educational videos represent a promising alternative, however their production typically requires significant time and financial resources. Video production using Generative AI technology may provide a solution to this problem.

Objective:

We used Natural Language Processing (NLP) to understand patients’ feedback provided in free text format on one of the seven Synthesia-generated patient educational videos created in collaboration with Roche UK and Hull University Teaching Hospitals Breast Cancer team, entitled "Breast Cancer Follow Up Programme".

Methods:

A survey was sent to 400 patients who had completed the breast cancer treatment pathway. 98 free-text responses were received for the question ‘Any comments or suggestions to improve [the video’s] contents?’. We applied and evaluated different NLP machine learning techniques to draw insights from these unstructured data, namely sentiment analysis, topic modelling, summarisation and TF-IDF word clouds.

Results:

Sentiment analysis showed that 80.6% of responses were positive or neutral, while negative comments were predominantly related to the AI avatar. Topic modelling using BERTopic with K-means clustering was found to be the most effective model, and identified four primary topics: breast treatment care, video content, digital avatar/narrator, and short responses with little/no content. The TF-IDF word clouds indicated positive sentiments about the treatment pathway (e.g., ‘reassured’, ‘faultless’) and video content (e.g., ‘informative’, ‘clear’), whereas the AI avatar was often described negatively (e.g., ‘impersonal’). Summarisation using the T5 model effectively created summaries of the responses by topic.

Conclusions:

This study demonstrates the success of NLP techniques in efficiently generating insights into patient feedback. Combining NLP methods resulted in clear visuals and insights, enhancing understanding of patient feedback. Analysis of free-text responses provided HUTH clinicians with deeper insights compared to quantitative Likert-scale responses. Importantly, the results validate the use of Generative AI in creating patient educational videos, highlighting its potential to address the challenges of costly video production and the limitations of traditional, often overwhelming educational leaflets. Despite the positive overall feedback, negative comments focused on the technical aspects of the AI avatar, indicating areas for improvement. We advocate that patients that receive AI avatar explanations are counselled that this technology is not replacing human, healthcare interactions. Future investigations will be necessary to confirm the ongoing effectiveness of such educational tools.