Previously submitted to: JMIR Medical Informatics (no longer under consideration since Apr 13, 2026)
Date Submitted: Aug 15, 2025
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Evaluation of Differential Diagnosis Accuracy Using a Customized GPT-Based Artificial Intelligence Model in the Emergency Department
ABSTRACT
Background:
Emergency departments (EDs) are high-pressure environments where rapid and accurate diagnosis is critical for patient outcomes. Artificial intelligence (AI)–based large language models (LLMs) are increasingly being explored as decision support tools to improve diagnostic accuracy, particularly when processing unstructured patient data.
Objective:
This study aimed to evaluate the diagnostic accuracy of differential diagnoses generated by a custom-trained with emergency medicine textbooks AI-based language model (Custom GPT) using unstructured symptom descriptions and physical examination findings of ED patients, and to assess associations with demographic and clinical variables.
Methods:
A retrospective, single-center study was conducted on 379 patients presenting to the ED. The model was trained using the Retrieval-Augmented Generation (RAG) method with a core emergency medicine textbook. For each case, the model generated one primary and four differential diagnoses. Primary diagnostic accuracy was compared with physician-confirmed diagnoses. Associations between accuracy and age, sex, triage category, organ system, and ED disposition were analyzed using statistical tests and correlation analysis.
Results:
The model achieved a primary diagnostic accuracy of 88.4%. Accuracy was highest in green triage patients (93.07%, p = 0.006) and lowest in red triage patients (75.31%, p < 0.001). Accuracy was significantly lower in cardiovascular system cases (p = 0.002) and in hospitalized patients compared to those discharged (p = 0.002). No significant association with age or sex was observed. Correlation analysis revealed a weak but significant positive correlation for green triage (R = 0.141; p = 0.006) and weak but significant negative correlations for red triage (R = –0.213; p < 0.001), hospitalization (R = –0.160; p = 0.002), and cardiovascular system cases (R = –0.162; p = 0.002).
Conclusions:
A Custom GPT model demonstrated high diagnostic accuracy and potential as a clinical decision support tool in the ED. However, caution is warranted when applying such models to complex, high-acuity cases.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.