Previously submitted to: JMIR AI (no longer under consideration since Sep 06, 2025)
Date Submitted: Jul 4, 2025
Open Peer Review Period: Jul 14, 2025 - Sep 8, 2025
(closed for review but you can still tweet)
NOTE: This is an unreviewed Preprint
Warning: This is a unreviewed preprint (What is a preprint?). Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn (a note "no longer under consideration" will appear above).
Peer review me: Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period (in this case, a "Peer Review Me" button to sign up as reviewer is displayed above). All preprints currently open for review are listed here. Outside of the formal open peer-review period we encourage you to tweet about the preprint.
Citation: Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
Final version: If our system detects a final peer-reviewed "version of record" (VoR) published in any journal, a link to that VoR will appear below. Readers are then encourage to cite the VoR instead of this preprint.
Settings: If you are the author, you can login and change the preprint display settings, but the preprint URL/DOI is supposed to be stable and citable, so it should not be removed once posted.
Submit: To post your own preprint, simply submit to any JMIR journal, and choose the appropriate settings to expose your submitted version as preprint.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Measuring Benchmarks for Differential Diagnosis Performance in Large Language Models, A Cross-Sectional Comparative Cohort Analysis
ABSTRACT
Background:
Differential diagnosis is an important skill, central to the clinical reasoning process. Large language models (LLMs) have demonstrated strong performance in this task among other clinical reasoning tasks. However, study instruments have not replicated the likely use of LLMs in clinical environments and LLM performance has been defined subjectively.
Objective:
The purpose of this paper was to assess LLM ability in generating a prioritized differential diagnosis by using objective benchmarks in response to a brief clinical vignette and comparing these benchmarks to experienced clinicians.
Methods:
This cross-sectional study utilized a survey with three clinical vignettes of 50 words or less. As a convenience sampling strategy in this study, the study group included the most recent versions of commonly used LLMs as of the study period. Each LLM participant was prompted, “Provide a differential diagnosis for this patient’s [chief complaint]. List potential diagnoses without additional explanations.” Zero-shot prompting was used to evaluate each LLM, and two iterations were measured. Outcomes included top slot diagnosis, differential concordance index, list length, time to completion, word count, and number of citable references included. A reference panel of experienced emergency and internal medicine clinicians allowed for comparison in this skill.
Results:
Ten different versions of LLMs were tested on each of the three vignettes. Top slot diagnosis concordance with the reference panel was 94.8% with eight of the ten LLM versions agreeing with the reference panel for 100% of the vignette iterations. Median list length per differential for LLMs (12, 10-15) was significantly greater than the reference panel lists (6, 5-9.5; X 5.95, p=0.000). LLM differential lists exhibited a variable concordance with reference panel lists with 20.1% of the lists containing unsupported items. LLMs provided differential lists in a significantly shorter amount of time as compared to the reference panel (X 4.99, p=0.001).
Conclusions:
Despite high top slot concordance with experienced clinician differential lists, LLM lists were significantly longer and exhibited variable global concordance which could lead to unnecessary testing and treatment. Future studies are needed to understand the implications of utilizing LLMs’ for diagnostic reasoning tasks in a clinical setting.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.