JMIR Publications

ABSTRACT

Background:

Information overload in electronic health records (EHRs) requires effective solutions to alleviate clinicians' administrative tasks. Automatically summarizing clinical text has gained significant attention with the rise of large language models (LLMs). While individual studies show strong optimism, a structured overview of the state-of-research is currently not available.

Objective:

We aim to present the current state of the art on clinical text summarization using large language models. In addition, we aim to evaluate the level of evidence in the current state of research and assess the reliability of performance findings for clinical application.

Methods:

This scoping review complies with the PRISMA-ScR guidelines. Literature published between January 1, 2019, and June 18, 2024, is identified from five databases: PubMed, Embase, Web of Science, IEEE Xplore, and ACM Digital Library. Data related to experimental design, evaluation methods, and other relevant factors are systematically collected and analyzed by three authors independently.

Results:

A total of 30 original studies are included in the analysis. The research landscape demonstrates a narrow research focus, predominantly centered on summarizing specific Chest X-Ray reports (26.7%), primarily involving patients in Intensive Care Units (50%) and data originating from U.S.-based institutions (63.3%). This focus aligns with the frequent reliance on the open-source MIMIC dataset (50%). While summarization methodologies vary, significant under-reporting exists in data input structure (50%), input source count (80%), summarization technique (33.3%), and deployment environment (83.3%). Heterogeneous evaluation frameworks hinder research integration, while reported strategies might fail to capture models’ translational value. In addition, ethical considerations are largely overlooked, with bias analysis entirely absent and only one study (3.3%) addressing risk analysis.

Conclusions:

While enthusiasm regarding large language models is warranted, our review highlights the importance of maintaining a measured, clear-sighted, and evidence-based approach. Limited scientific evidence stems from under-reported experimental designs and heterogeneous evaluation frameworks across studies. Prudent and carefully monitored use of these models in clinical settings is therefore crucial. To advance the field, future research should emphasize transparency to enable research integration and build on prior work. Moreover, evaluation frameworks must prioritize the translational value of these models to more effectively assess their performance, applicability, and alignment with ethical standards in clinical settings.