JMIR Publications

ABSTRACT

Background:

Generative Artificial Intelligence (AI) tools are emerging as a promising solution to streamline and improve clinical documentation. Discharge instructions are a crucial part of Emergency Department (ED) care, but due to time constraints, they may lack key elements like anticipatory guidance or return precautions. AI may expedite and improve discharge instruction writing by providing comprehensive drafts for clinicians’ review.

Objective:

Evaluate the quality of AI-generated vs. handwritten discharge instructions, as well as a commercial AI tool’s ease of use and providers’ readiness to adopt similar technologies in clinical workflows.

Methods:

Surveys were administered via Qualtrics. Participants were asked about their opinions of AI, to rate three discharge summaries, one of which was AI-generated, and to draft discharge instructions with ChatGPT 4.0 using a one-shot method. Additive Likert scores were used to assess enthusiasm and familiarity with AI and the quality of all discharge instructions. Quantitative data was analyzed in GraphPad Prism 10.2. Two investigators performed narrative synthesis to identify common themes in short-answer responses and ChatGPT prompts/outputs. Ngram analysis examined free-text elements for keyword frequency.

Results:

76 clinicians initiated and completed at least one part of the survey. Participants’ familiarity with and enthusiasm for generative AI did not correlate with their provider role or years in practice, and familiarity with AI did not covary with enthusiasm (p>0.05). The AI-generated discharge summary received higher quality scores than either form of handwritten discharge instructions (p<0.01), and fared especially well compared to the brief handwritten discharge summary (p<0.0001). The differences in quality score between discharge instructions were most pronounced amongst residents. The AI-generated discharge summary was consistently attributed to AI (≥88%). Conversely, The brief handwritten discharge summary was consistently identified as not AI (≥86%). All participants who completed part 3 were satisfied with their AI-generated discharge instructions, and only 26% re-prompted ChatGPT. Quality scores did not differ between providers who prompted once or twice (p>0.05). NGram analysis showed that ChatGPT outputs consistently used the same keywords and provided section headings. On the other hand, initial prompts usually contained common sense keywords like “discharge instructions” and “chest pain,” but re-prompts were highly variable.

Conclusions:

Generative AI has the potential to help providers write discharge instructions more efficiently and thoroughly. An AI-generated discharge summary was rated higher in quality than two forms of handwritten discharge instructions by ED clinicians, and ED clinicians indicated satisfaction when generating their own discharge instructions using a commercially available generative AI tool. ED clinicians were also able to reliably differentiate between handwritten and AI-generated discharge instructions, underscoring the need for further research of all key stakeholders’ perceptions on quality and safety of an AI-assisted documentation workflow.