JMIR Publications

ABSTRACT

Background:

The increasing documentation burden on physicians is a significant contributor to burnout and decreases in care quality. Artificial Intelligence (AI) has been proposed as a solution to reduce documentation burden in clinical care, but there is very limited data on its use in the inpatient and intensive care unit (ICU) environments.

Objective:

This study explores the application of AI-assisted ambient scribes to improve documentation efficiency and clinician satisfaction during ICU rounds. We focus on customizing the prompt for large language models (LLMs) to generate and evaluate daily progress notes from transcripts of ICU multidisciplinary rounds.

Methods:

Our project is divided into two phases. In the first phase, a transcript of an audio recording of a randomly selected ICU rounds case was used to evaluate and improve the prompt for the LLM iteratively. Multiple models (five) were used in phase one, and the best-performing model (M1, based on the highest accuracy as determined by the team) was selected for the next phase. In the following phase, five cases were selected and evaluated using the improved prompt and two separate models (M1 from phase 1 and new model M6, which is a technological upgrade of the M1). Metrics included accuracy and error percentages. Additionally, error severity and readability were assessed using the Harm scale (adapted for potential harm risk) from the Agency for Healthcare Research and Quality Physician Documentation Quality Instrument–9 (PDQI-9), respectively.

Results:

Iterative improvements to the prompt resulted in increased accuracy and a reduction in errors during Phase 1. In Phase 2, M1 and M6 achieved accuracies of 70% and 82%, respectively (p<0.0001). Errors of omission were most common (60.5±5.5%), followed by partial errors (30.8±4.1%) and then errors of commission (8.6±2.9%). The error severity of both models was low (µ = 0.62 vs. 0.58, p = 0.28), with most errors categorized as having potential for No Harm to Low Harm. Both models performed well on the PDQI-9 assessment, with the M6 model outperforming the M1 (35.8 vs. 38.3, p = 0.015).

Conclusions:

Our findings demonstrate the feasibility of integrating AI-assisted scribes for ICU documentation. Both prompt improvement and technological advancements in LLM are noted to be helpful. This study lays the groundwork for future research into AI applications in ICU settings, paving the way for broader improvements in healthcare documentation.