JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) have emerged as promising tools for enhancing public access to medical information, particularly for chronic diseases such as atherosclerotic cardiovascular disease (ASCVD). However, their effectiveness in patient-centered health communication remains underexplored, especially in multilingual contexts.

Objective:

This study aimed to compare the performance of state-of-the-art LLMs in generating reliable health information for ASCVD patients across English and Chinese, to evaluate their suitability for public health communication and patient education.

Methods:

We evaluated the performance of three advanced LLMs—Deepseek-R1, ChatGPT-4o, and Google Gemini—in generating accurate, complete, and comprehensible responses to 25 open-ended ASCVD-related questions in both English and Chinese. Each question was submitted five times per model per language, yielding 750 responses. Responses were blindly assessed by three board-certified cardiologists using standardized criteria.

Results:

Deepseek-R1 achieved the highest "Good response" rates (96% in both English and Chinese), outperforming ChatGPT-4o (84%) and Gemini (48% in English, 68% in Chinese). It demonstrated significantly superior accuracy and completeness across both languages (p < 0.01), while all models performed comparably in comprehensibility. No significant differences were observed between English and Chinese performance for Deepseek-R1 and ChatGPT-4o; Gemini showed greater variability but without statistical significance. Diagnostic and definitional questions yielded stronger responses than treatment or prevention topics.

Conclusions:

Deepseek-R1 showed robust and consistent performance in patient-facing ASCVD queries across languages, highlighting the potential of open-source LLMs in promoting digital health literacy and equitable access to chronic disease information. These findings support the cautious integration of LLMs into public health communication strategies, alongside expert oversight and continued validation.