JMIR Publications

ABSTRACT

Background:

Radiology reports serve as the definitive clinical record of imaging diagnoses, with their accuracy being directly critical to patient safety. However, amidst escalating workloads, manual review remains inadequate for capturing subtle errors in reports on a large scale. Although large language models (LLMs) have demonstrated considerable promise in medical text processing, previous studies have predominantly been confined to single-modality or synthetic datasets.

Objective:

This study aimed to evaluate the feasibility, accuracy, and generalizability of general-purpose large language models in assisting with radiology report quality assurance across multimodal (X-ray, CT, MRI), multi-anatomical, and multi-center clinical settings, with head-to-head comparisons conducted against radiologists of varying seniority.

Methods:

The multimodal (X-ray, CT, MRI) and multisite (head, chest, abdomen, spine, and knee) radiology reports from Center 1 (Mar–May 2025) were retrospectively reviewed, using a clinician-annotated five-category error taxonomy (Type 1: interpretive; Type 2: factual). After benchmarking five LLMs (Qwen 3, GPT-5.4 Pro, Gemini 3.1 Pro, Llama 4, DeepSeek-V3.1), the optimal LLM was selected for full evaluation, including head-to-head comparison with three radiologists (3–10 years’ experience) on 300 reports, clinician-rated reasoning assessment (5-point Likert scale), and cross-center generalizability testing on reports from a second institution.

Results:

In the primary cohort, 4,431 of 95,582 reports contained expert-confirmed errors. The selected model achieved sensitivity 0.99, specificity 0.99, positive predictive value 0.90 and F1 score 0.94 for report-level error detection. Performance was robust across modalities and anatomical regions, with strongest performance for factual errors. In head-to-head testing, Qwen 3.0 achieved higher accuracy than radiologists (0.96 vs 0.45-0.66; P<0.001) and processed reports faster (5.8 vs 55.8-67.8 s per report). External validation in 44 839 reportspreserved high balanced accuracy (0.94 [0.92-0.95]).

Conclusions:

This study demonstrates that large language models can serve as highly efficient and reliable auxiliary tools for radiology report quality assurance. In detecting report errors, particularly unambiguous factual errors, they exhibit accuracy and processing speeds that surpass those of human experts, alongside robust cross-center generalizability. These findings provide compelling evidence for the implementation of automated radiology report review strategies in real-world clinical environments.