JMIR Publications

ABSTRACT

Background:

Emergency departments (EDs) face significant challenges due to overcrowding, prolonged waiting times, and staffing shortages, leading to increased strain on healthcare systems. Efficient triage systems and accurate departmental guidance are critical to alleviating these pressures. Recent advancements in Large Language Models (LLMs), such as ChatGPT, offer potential solutions for improving patient triage and outpatient department selection in emergency settings.

Objective:

The study aims to assess the accuracy, consistency, and feasibility of GPT-4 based ChatGPT models (GPT-4o and GPT-4-Turbo) for patient triage using the Modified Early Warning Score (MEWS) and evaluate GPT-4o’s ability to provide accurate outpatient department guidance based on simulated patient scenarios.

Methods:

A two-phase experimental study was conducted. In phase one, two ChatGPT models (GPT-4o and GPT-4-Turbo) were evaluated for MEWS-based patient triage accuracy using 1,854 simulated patient scenarios. Accuracy and consistency were assessed before and after prompt engineering. In phase two, GPT-4o was tested for outpatient department selection accuracy using 264 scenarios sourced from the Chinese Medical Case Repository. Each scenario was independently evaluated by GPT-4o three times. Data analyses included Wilcoxon tests, Kendall correlation coefficients, and logistic regression.

Results:

In the first phase, ChatGPT’s triage accuracy, based on the MEWS, improved following prompt engineering. Interestingly, GPT-4-Turbo outperformed GPT-4o, achieving an accuracy of 100% compared to GPT-4o's 96.2%, despite GPT-4o initially showing better performance prior to prompt engineering, suggesting GPT-4-Turbo may be more adaptable to prompt optimizations. In the second phase, GPT-4o, with a superior performance on emotional responsiveness compared to GPT-4-Turbo, demonstrated an overall guidance accuracy of 92.63% (95% CI, 90.34%, 94.93%), with the highest accuracy in internal medicine (93.51%, [95% CI, 90.85%, 96.17%]). and the lowest in general surgery (91.46%, [95% CI, 86.50%, 96.43%]).

Conclusions:

ChatGPT demonstrates promising capability for supporting patient triage and outpatient guidance in EDs. GPT-4-Turbo showed greater adaptability to prompt engineering, whereas GPT-4o exhibited superior responsiveness and emotional interaction, essential for patient-facing tasks. Future studies should explore real-world implementation and address identified limitations to enhance ChatGPT’s clinical integration.