JMIR Publications

ABSTRACT

Background:

Mass casualty incidents require rapid decisions under high pressure, and experts are often not the first on site. Large Language Models (LLM) like ChatGPT could serve as a readily available expert system to aid decision-making. There is a lack of scientific studies investigating how the output depends on systematically modified prompts and the version model.

Objective:

This study aims to compare the triage accuracy of five ChatGPT versions using 30 simulated pre-hospital disaster cases and to examine to what extent systematically extended prompts affect the outcome. Additionally, over- and undertriage rates were investigated, as well as the creation of a meaningful ranking of all patients.

Methods:

30 fictious case vignettes were triaged using five different versions of ChatGPT: GPT-3.5, GPT-4, GPT-4o, GPT-o1, and GPT-5. Initially, an openly formulated prompt was used, requesting triage, including an explanation. Subsequently, ChatGPT was asked to use the mSTART, then the PRIOR algorithm. Finally, the prompt was extended to include a detailed description of the respective algorithm. Each constellation of case vignette, model version, and prompt was repeated five times separately, and a final response was selected for further analysis based on majority vote. Accuracy, over-, and undertriage were examined as outcomes and paired comparisons with and without a detailed description of an algorithm were tested for significance using the exact McNemar test. Finally, ChatGPT was asked to create a treatment ranking of multiple patients.

Results:

With an openly formulated prompt, ChatGPT achieved accuracy values ranging from 56.7% (GPT-4o) to 90% (GPT-5). Mentioning an algorithm did not result in consistent improvements. A detailed description of the PRIOR algorithm considerably increased accuracy. ChatGPT 5, prompted to use the PRIOR algorithm, achieved the best results without a detailed description, reaching 93.3% (28/30). ChatGPT 4 and GPT-4o, with the specific PRIOR algorithm included in the prompt, achieved the highest overall accuracy, at 96.7% (29/30) each. These two cases showed a significant improvement in the exact McNemar (GPT-4: P = .008; GPT-4o: P = .016). While older versions tend towards undertriage, modern versions showed a more balanced rate. When tasked with ranking ten patients, rate of correctness ranged from 80% to 86.7%. When all 30 patients were ranked together, only ChatGPT 5 remained equally good at 86.7%, while GPT-3.5 correctly classified 36%.

Conclusions:

Triage performance using ChatGPT depends on the model version and can be influenced by specifically tailored prompts. Newer models achieve higher accuracy but benefit less from a detailed algorithm description. However, this approach has shown a significant increase in accuracy with selected older versions. Despite good triage accuracy, LLMs may currently be more suitable as supervised decision support than autonomous triage systems. To further assess triage competence, validation through realistic simulations is indispensable.