JMIR Publications

ABSTRACT

Background:

As the cancer survivor population grows and new information technologies become widespread in Hong Kong SAR, cancer survivors increasingly seek ongoing care and support information from large language models (LLMs). While these tools provide immediate conversational responses, they carry substantial risks of generating inaccurate, unsafe, or generic medical advice. Evaluating LLM response performance is particularly critical in Hong Kong's bilingual healthcare context.

Objective:

This study aimed to evaluate the medical accuracy, safety, understandability, and readability of four popular commercial LLMs answering cancer survivorship questions in English and Traditional Chinese. Furthermore, to determine whether a structured prompting strategy effectively improves response performance.

Methods:

A two-phase mixed-methods study was conducted. In Phase I, 25 expert-curated questions spanning five survivorship domains were input into four LLMs: Google Gemini 3.1 Pro, HK Chat-0.6.2, DeepSeek-V3.2, and Kimi-K2.5. A Delphi expert panel evaluated the baseline responses for medical accuracy using a 5-point scale and assessed safety via binary categorization. Understandability was measured using the Patient Education Materials Assessment Tool (PEMAT). Readability was assessed via the Flesch-Kincaid Grade Level for English and lexical richness for Traditional Chinese. In Phase II, experts developed an optimized CRAFT (Context, Role, Audience, Format, Task, Tone) prompt, and the resulting LLM responses were comparatively validated against baseline outputs using paired statistical analyses.

Results:

Baseline evaluations identified Google Gemini 3.1 Pro as the most accurate model in English (mean 3.99/5) and Traditional Chinese (4.21/5). Unconstrained models demonstrated language-dependent safety vulnerabilities: English errors (up to 24.0%) centered on service mismatches and definitive interpretations, whereas Traditional Chinese errors (up to 20.0%) involved prescriptive disease management and unverified adjunctive therapies. Highly readable models generally exhibited lower accuracy. Applying the CRAFT prompt to Gemini 3.1 Pro significantly improved medical accuracy across both languages (English: 4.68, P < .001; Traditional Chinese: 4.66, P < .001). This prompt optimisation resolved the specific unsafe English baseline responses, ensuring safety compliance without compromising patient understandability.

Conclusions:

LLMs exhibit significant, language-divergent clinical safety vulnerabilities and an inherent trade-off between medical accuracy and readability when answering cancer survivorship queries. Expert-designed CRAFT prompting effectively addresses these limitations by correcting unsafe medical directives and enhancing accuracy, preserving understandability. Safe deployment of LLMs in multilingual oncology contexts requires strict validation combined with structured, role-restrictive constraints.