JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are increasingly used by students, junior clinicians and patients for medical learning and health information, yet their specialist knowledge within narrow, clinically heterogeneous disease domains is difficult to assess from standardised multiple-choice benchmarks. Such benchmarks predominantly measure factual recall in English and do not capture the integrated, multi-element specialist knowledge expected in neurology training. Tremor is a common, clinically heterogeneous movement-disorder presentation for which Chinese-language, expert-validated evaluation tools are lacking.

Objective:

To evaluate the tremor-related specialist knowledge of three mainstream Chinese LLMs using an expert-designed six-dimension question matrix, pre-specified per-question answer keys, and blinded independent expert scoring, and to provide a transparent, reusable assessment instrument for medical education and model selection.

Methods:

Three commercial Chinese-context LLMs (DeepSeek-R1-671B, Moonshot-v1-128k-vision-preview, Doubao-1.5-pro-256k) were tested under identical single-turn, zero-context settings on 36 integrated questions spanning six clinical reasoning dimensions (pathogenesis, risk factors, clinical manifestations, diagnosis, treatment/prevention, prognosis). Questions were developed by five movement-disorders chief physicians through a modified Delphi process; a written answer key specifying required elements, critical omissions and safety-critical items was finalised for every question before scoring. Three different chief physicians, blinded to model identity, independently scored all de-identified, randomised responses on a 0–10 professional-competence rubric (108 model–question pairs per model) and, for the six treatment questions, a 0–10 treatment-safety rubric. A distilled model (DeepSeek-R1-Distill-Llama-70B) was tested identically as an exploratory comparison. Reliability was assessed with intraclass correlation coefficients (ICC); model comparisons used the Friedman test with Dunn–Bonferroni post-hoc tests and paired Wilcoxon signed-rank tests.

Results:

Professional-competence scores differed across models (Friedman χ²=65.15, df=2, P<.001). DeepSeek-R1-671B scored highest (mean 9.13/10, SD 0.23 across questions), followed by Doubao-1.5-pro-256k (6.80, SD 1.28) and Moonshot-v1-128k (4.88, SD 0.88); all pairwise differences remained significant after Bonferroni adjustment (all P≤.002). On safety, DeepSeek scored highest (8.61, SD 0.92), followed by Doubao (7.94, SD 0.94) and Moonshot (5.33, SD 1.78); one Moonshot response received an automatic zero for an unsafe medication recommendation. Reliability of the averaged scores was high (ICC(2,k)=0.94, 95% CI 0.91–0.96; single-rater ICC(2,1)=0.83, 95% CI 0.78–0.88). DeepSeek-R1-671B outscored the 70B model (5.82, SD 1.31) on all 36 questions (Wilcoxon P<.001).

Conclusions:

Under controlled single-turn testing, one flagship Chinese LLM provided accurate, expert-rated specialist answers across the tremor knowledge pathway, whereas the other two were less consistent and included a safety failure. The instrument measures static, context-free specialist knowledge relevant to medical education and assisted reference use, not interactive diagnostic reasoning; clinical-vignette and multi-turn assessment is a distinct next step. The six-dimension matrix, per-question answer keys and blinded rubric offer a reusable, reproducible framework for domain-specific LLM evaluation in education. Clinical Trial: Not applicable (cross-sectional survey study).