Currently submitted to: Journal of Medical Internet Research
Date Submitted: Oct 14, 2026
Open Peer Review Period: Oct 15, 2026 - Dec 10, 2026
(currently open for review)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Large Language Model Assistance for Diagnostic Reasoning in Rare Neurological Disease: Randomized Controlled Trial
ABSTRACT
Background:
Rare neurological diseases are frequently misdiagnosed, particularly among early-career physicians. Although large language models (LLMs) perform well on medical examinations, their effect on clinicians’ diagnostic reasoning remains unknown.
Objective:
To assess the effect of LLM assistance on diagnostic reasoning among resident and attending neurologists evaluating rare neurological diseases.
Methods:
A randomized, single-blind trial at 7 hospitals in China. Neurologists were recruited May 25 to June 10, 2026, and completed the assessment remotely or in person on June 14-15, 2026. Participants were randomly assigned 1:1, stratified by career stage, to complete 8 written cases of rare neurological disease using conventional resources with or without DeepSeek-V3. A nonrandomized exploratory arm ran the same cases through the model alone. The primary outcome was the case-level total rubric score, expressed as a percentage of points available. Secondary outcomes included the score in each of 5 rubric domains, whether the reference diagnosis was ranked first, whether it was included in the differential diagnosis, and completion time. All analyses followed the intention-to-treat principle.
Results:
Sixty physicians were randomized (30 per group) and included in the primary analysis (480 observations). Model-estimated mean scores were 81.4% (95% CI, 76.5-86.4) with LLM assistance vs 55.7% (95% CI, 50.8-60.6) with conventional resources, a difference of 25.7 percentage points (95% CI, 18.7-32.7; P<.001). The reference diagnosis was more often ranked first with LLM assistance (61.5% vs 29.9%; difference, 31.7 percentage points; 95% CI, 22.5-40.8) and more often included in the differential diagnosis (91.3% vs 58.7%; difference, 32.6 percentage points; 95% CI, 23.9-41.3) (both P<.001). Completion time was shorter with LLM assistance (median, 32.0 [IQR, 22.0-49.1] vs 57.6 [IQR, 45.2-77.9] minutes; difference, −25.6 minutes; 95% CI, −44.1 to −8.2; P<.001). Post hoc adjustment for response length attenuated the primary difference to 14.5 percentage points (95% CI, 6.5-22.4; P<.001). The model alone scored a mean of 81.8% (SD, 6.4).
Conclusions:
In this trial, access to a freely available LLM improved diagnostic reasoning and shortened completion time among resident and attending neurologists evaluating rare neurological diseases. Whether this benefit extends to real clinical encounters warrants further investigation. Clinical Trial: Chinese Clinical Trial Registry; ChiCTR2600125237. https://www.chictr.org.cn/bin/project/edit?pid=274923
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.