JMIR Publications

ABSTRACT

Background:

The use of large language models (LLMs) in radiology is expanding rapidly. Yet, a comprehensive evaluation of their applications, performance, and limitations is lacking

Objective:

To map current applications of large language models (LLMs) in radiology, evaluate their performance across key tasks, and identify current limitations and future research directions.

Methods:

A scoping review was conducted following Arksey and O'Malley's framework and the PRISMA-ScR guidelines. Three databases—SCOPUS, PubMed, and IEEE Xplore—were searched for studies between January 2022 and June 2024. Eligible studies included empirical evaluations of LLMs applied to radiological data or workflows. Exclusion criteria included commentaries and purely technical model proposals without evaluation. No formal risk of bias assessment was performed. Results were synthesized narratively and tabulated.

Results:

Sixty-seven studies were included. GPT-4 was the most common model (53.7%), and radiology reports were the most frequently used data source (39%). Applications clustered into three themes: (1) decision support, (2) report generation, and (3) workflow optimization. While performance in linguistic tasks was high (e.g., report simplification accuracy >94%), diagnostic accuracy varied widely (16–86%), often limited by dataset bias and lack of contextual reasoning.

Conclusions:

LLMs have demonstrated utility in streamlining radiological documentation and augmenting non-diagnostic tasks. However, variability in diagnostic performance and lack of external validation remain concerns. Further research into domain-specific and multimodal models, as well as prospective evaluations, is warranted.