JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are increasingly used for health information, but COPD-related questions often involve safety-sensitive decisions, including treatment escalation, oxygen therapy, and exacerbation management.

Objective:

To compare six public-facing LLM chatbots in answering COPD-related public health questions across safety, accuracy, empathy, information quality, source transparency, and readability.

Methods:

A 55-question English COPD question bank was developed from an initial 108-question pool using web-based retrieval, Google Trends, PubMed-informed topic review, patient education resources, and expert screening. ChatGPT 5.5 Thinking, Gemini 3.1 Pro, Copilot Deep Thinking, DeepSeek Expert Mode, Doubao Pro Mode, and Qwen 3.6-Plus were queried once per question in separate or cleared sessions. Five blinded raters assessed responses against a GOLD 2026-centered reference standard. Manual scores and readability indices were analyzed using agreement statistics, paired model comparisons, effect sizes, and qualitative review of unsafe responses.

Results:

The 55 questions yielded 330 responses. Unsafe responses occurred in every model: ChatGPT, Gemini, Copilot, and DeepSeek each generated 4 unsafe responses (7.3%), Qwen generated 5 (9.1%), and Doubao generated 6 (10.9%). Safety differences were not statistically significant (Cochran’s Q = 1.909, df = 5, P = 0.862). Accuracy, empathy, information quality, and readability differed significantly across models. DeepSeek and Qwen performed strongly on accuracy, DISCERN, and GQS, whereas Gemini generated the most readable responses. Unsafe patterns involved medication escalation, oxygen therapy, exacerbation recognition, antibiotics, and oral corticosteroids.

Conclusions:

LLMs may support general COPD education, but all evaluated models produced potentially unsafe advice in clinically actionable domains and should not replace individualized clinician guidance.