JMIR Publications

ABSTRACT

Background:

Large language model (LLM)-based chatbots may help disseminate scalable, evidence-informed health information during postearthquake crises, when health care infrastructure and professional consultation may be disrupted. However, the quality, readability, reliability, and short-term stability of artificial intelligence (AI)-generated guidance across platforms, languages, and reasoning modes remain insufficiently characterized.

Objective:

This study aimed to compare the quality, readability, and temporal stability of responses generated by four mainstream LLM-based chatbots to expert-reviewed postearthquake health frequently asked questions (FAQs), with attention to language and reasoning-mode differences.

Methods:

An expert-reviewed bank of 25 postearthquake health FAQs was submitted to four platforms (ChatGPT-5.5, Gemini 3.1 Flash, DeepSeek-V4, and Doubao) in English and Chinese under standard and platform-specific reasoning-enhanced ("Thinking") modes. In total, 600 single-turn responses were independently evaluated by two blinded disaster medicine experts using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P), Global Quality Score (GQS), and modified DISCERN (mDISCERN). Readability indices were calculated for English and Chinese responses. Temporal stability was assessed using two one-sided tests (TOST) for equivalence and Bland-Altman analysis, and semantic stability was assessed using cosine similarity and ROUGE-L text overlap.

Results:

Pooled interrater reliability was excellent in the primary Chinese standard-mode Round 1 subset (all pooled intraclass correlation coefficients [ICCs] for the ICC[2,k] model >0.90), and updated platform-stratified estimates were good to excellent across metrics (ICC[2,k] range 0.892-0.980). In the primary comparison (Chinese, standard mode, Round 1), significant overall platform effects were observed for all evaluated quality metrics (all P<.001), with ChatGPT-5.5 and Gemini 3.1 Flash generally ranking above Doubao and DeepSeek. Language effects were bidirectional: Chinese responses scored higher for understandability and actionability, whereas English responses scored higher for information reliability as measured by mDISCERN. The reasoning-enhanced "Thinking" mode did not consistently improve response quality. Instead, it decreased practical actionability across several platforms (eg, GPT, mean difference -0.62; P<.001), suggesting a potential trade-off between completeness and actionability. Temporal stability was generally acceptable at the group level, but individual-level variability remained nonnegligible; English outputs met equivalence criteria more often, whereas limits of agreement were metric- and platform-dependent. English responses from all platforms exceeded the recommended eighth-grade reading level.

Conclusions:

LLM-based chatbots show promise for postdisaster health communication, but their usefulness depends on platform, target language, and reasoning-mode configuration. Reasoning-enhanced modes may reduce the concise actionability needed for emergency instructions. Validation against authoritative public health guidance and testing with lay users are needed before real-world deployment in public health emergencies.