Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Nov 30, 2023
Open Peer Review Period: Nov 30, 2023 - Jan 26, 2024
Date Accepted: Mar 10, 2024
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
ChatGPT-4 Performance in Cardiac Arrest & Bradycardia Simulations Using American Heart Association (AHA) Advanced Cardiac Life Support (ACLS) Guidelines: An Exploratory Study
ABSTRACT
Background:
ChatGPT-4 is the most advanced large language model (LLM) to date, with prior iterations passing medical licensing exams, providing clinical decision support and improved diagnostics. Although limited, past studies of ChatGPT’s performance found the AI could pass American Heart Association (AHA) Advanced Cardiac Life Support (ACLS) exams with modifications. ChatGPT-4’s accuracy has not been studied in more complex clinical scenarios. As heart disease and cardiac arrest remain leading causes of morbidity and mortality in the US, finding technologies that help increase adherence to ACLS algorithms, which improves survival outcomes is critical.
Objective:
Our study examines the accuracy of ChatGPT-4 in following ACLS guidelines for bradycardia and cardiac arrest.
Methods:
We evaluated the accuracy of ChatGPT-4's responses to two simulations based on the 2020 AHA ACLS guidelines with three primary outcomes of interest: the mean individual step accuracy, the accuracy score per simulation attempt, and the accuracy score for each algorithm. For each simulation step, ChatGPT was scored for correctness (1 point) or incorrectness (0 points). Each simulation was conducted 20 times
Results:
ChatGPT’s average accuracy for cardiac arrest was 69% (IQR: 7; 67% - 75%) and for bradycardia was 43% (IQR: 17; 33% - 50%). We found ChatGPT’s outputs varied despite consistent input, the same actions were persistently missed, repetitive overemphasis hindered guidance, and erroneous medication information was presented.
Conclusions:
This study highlights the need for consistent and reliable guidance to prevent potential medical errors and optimize the application of ChatGPT to enhance its reliability and effectiveness in clinical practice.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.