JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) show promise in automatically detecting errors in radiology reports, but their performance remains insufficiently validated in large-scale, real-world clinical datasets.

Objective:

We aim to systematically evaluate the performance of LLMs in detecting and correcting errors in Chinese radiology reports derived from authentic clinical data.

Methods:

A large-scale dataset of Chinese radiology reports containing real clinical practice-generated errors was retrospectively collected between January 2023 and June 2024 at a single institution. Reports were randomly divided into test and internal validation sets. An additional 200 English-language reports from MIMIC-III were used for external validation. Eight human readers and eight widely adopted LLMs enhanced by prompt engineering were tasked with error detection. Overall and subgroup detection performance and reading time were evaluated. Correction suggestions from LLMs were reviewed by a senior radiologist.

Results:

A total of 1,363 radiology reports containing 1,551 errors were analyzed. DeepSeek-R1 achieved the highest detection rate of 89 %, 83 %, and 94 % across the test, internal, and external validation sets. On the internal validation set, its detection performance was comparable to that of radiologists and significantly higher than that of non-radiologists and non-physicians. Moreover, both DeepSeek-R1 and Claude-3.5-sonnet exhibited strong error correction capabilities, with correction accuracies of 95% and 91%, respectively.

Conclusions:

Enhanced LLMs, particularly DeepSeek-R1, demonstrated robust error detection and correction in real-world Chinese radiology reports, which support their clinical utility for automated quality assurance and integration into workflows to improve reporting accuracy and efficiency.