JMIR Publications

ABSTRACT

Background:

Transfer of patients with suspected ST-elevation myocardial infarction (STEMI) requires timely and robust communication, but clinical uptake of potentially useful information from physician-to-physician phone calls authorizing transfer is low, at least in part due to relative inaccessibility.

Objective:

We sought to evaluate whether large language models (LLMs) can effectively summarize these transfer calls, which could facilitate access to valuable information for receiving physicians.

Methods:

STEMI transfer calls for which our institution was the receiving facility were transcribed and summarized by OpenAI Whisper and ChatGPT software, respectively. Each summary was reviewed by two of seven independent physician-raters. Summaries were rated using the Likert Scale applied to an eight-domain framework adapted from the Physician Document Quality Instrument. In addition, we calculated raw inter-rater agreement for each domain across all summaries.

Results:

Exact agreement between raters was 62%, and 90% of paired ratings were within one point of each other. The mean rating of all summaries across all domains was 4.6 out of 5. Sixty-five percent of summaries had mean ratings of >4/5 on all eight domains. The ‘useful’ (4.84/5) and ‘consistent’ (4.85/5) domains were highest rated, and the ‘thorough’ (4.42/5) and ‘hallucination’ (4.40/5) domains were lowest rated.

Conclusions:

This proof-of-concept study suggests that LLMs can generate accurate and pertinent summaries of interhospital transfer calls for STEMI patients. Areas for future investigation include assessing the impact of these summaries on clinical outcomes, expanding to other patient populations, and applying this approach to additional clinical scenarios in which physician-to-physician communication plays a significant role.