Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Currently submitted to: Journal of Medical Internet Research

Date Submitted: Jun 12, 2026

Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.

Human–LLM Dialogue Improves Diagnostic Accuracy in Emergency Care: Controlled Crossover Study

  • Burcu Sayin; 
  • Ngoc Vo Hong; 
  • Ipek Baris Schlicht; 
  • Jacopo Staiano; 
  • Pasquale Minervini; 
  • Sara Allievi; 
  • Nicola Susca; 
  • Nicola Osti; 
  • Alberto Maino; 
  • Vito Racanelli; 
  • Andrea Passerini

ABSTRACT

Background:

Clinical decision-making in emergency medicine requires rapid, accurate diagnoses under uncertainty. Large language models (LLMs) have shown diagnostic potential, but evidence for their benefit in real physician workflows remains limited.

Objective:

This study evaluated whether MedSyn, a physician–LLM dialogue system, improves diagnostic accuracy in live emergency care workflows, and examined how benefit varies with case difficulty and physician expertise.

Methods:

Seven physicians (three senior internists, four residents) completed unassisted (baseline) and AI-assisted (interactive) sessions across 52 MIMIC-IV cases stratified by difficulty (Easy, Medium, Hard). Diagnostic performance was measured using automated fuzzy-matching metrics (any-match accuracy, exact-match accuracy, diagnosis-set F1) and blinded expert evaluation. Between-condition comparisons used paired bootstrap resampling (20,000 replicates, seed 42).

Results:

Interactive support significantly improved overall diagnostic accuracy: difficulty-standardised any-match accuracy improved by 0.156 (P < .001; d = 1.25). Residents showed the largest gains, with F1 improving by 0.138 (P < .001; d = 0.94). Blinded evaluation revealed residents’ Hard-case mean correctness rose from 0.589 to 0.734 (+0.145); difficulty-standardised completely-correct rates confirmed a medium effect (Δ = 0.092; P = .07; d = 0.47). Automated and blinded manual assessment showed only modest agreement (Cohen's κ = 0.148), with 45% of clinically correct diagnoses receiving no automated credit; reported F1 improvements are therefore conservative estimates. Cross-expertise concordance increased significantly (Δ = 0.142; P < .001; d = 1.48). Dialogue analysis revealed expertise-dependent strategies: seniors employed targeted, hypothesis-driven queries while residents used broader exploratory questioning.

Conclusions:

Interactive LLM assistance meaningfully enhances diagnostic accuracy, particularly for resident physicians on complex cases. The expertise-stratified benefit profile, large effect sizes, and significant convergence in cross-expertise concordance suggest that adaptive LLM deployment, targeting high-complexity cases and less experienced clinicians, is a promising and evidence-grounded strategy for emergency diagnostic support.


 Citation

Please cite as:

Sayin B, Hong NV, Baris Schlicht I, Staiano J, Minervini P, Allievi S, Susca N, Osti N, Maino A, Racanelli V, Passerini A

Human–LLM Dialogue Improves Diagnostic Accuracy in Emergency Care: Controlled Crossover Study

JMIR Preprints. 12/06/2026:104528

DOI: 10.2196/preprints.104528

URL: https://preprints.jmir.org/preprint/104528

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.