JMIR Publications

ABSTRACT

Background:

Although large language models (LLMs) show promise for radiology report quality control (QC), their performance and efficiency remain inadequate for routine clinical workflows.

Objective:

To develop and evaluate a privacy-preserving Radiology Report Quality Control LLM (RadQC-LLM) with efficient inference, and to validate its performance on multicenter real-world liver MRI report datasets and its clinical value through a multi-reader study.

Methods:

We fine-tuned the Unisound Shanhai Medical LLM using 1,917 expert-annotated reports to develop a RadQC-LLM. Evaluation comprised two stages: (1) benchmarking against general-purpose LLMs (ChatGPT 5.0, DeepSeek V3.2 Reasoner, and DeepSeek R1) on a balanced error-type dataset of 624 liver MRI reports with manually inserted errors; and (2) validation on a multicenter real-world dataset of 3,000 liver MRI reports retrospectively collected from four centers, benchmarked against the locally deployable DeepSeek R1. All reports were de-identified. A randomized crossover multi-reader study assessed model assistance across radiologists with varying experience. Permutation tests and Wilcoxon signed-rank tests were used for statistical comparisons.

Results:

In the balanced error-type dataset, RadQC-LLM achieved the highest report-level F1-score (0.812), significantly outperforming DeepSeek-R1 (0.730) and GPT-5 (0.765; both p<0.01), with no significant difference from DeepSeek-V3.2 (0.801; p=0.249), and achieved a superior micro F1-score at the error-type level (0.624) compared with DeepSeek-R1 (0.405), DeepSeek-V3.2 (0.561), and GPT-5 (0.505; all p<0.01). In the multicenter real-world dataset, RadQC-LLM achieved higher report-level and type-level F1-scores (0.603 and 0.373) than DeepSeek R1 (0.566 and 0.239; both p<0.01). The average processing time per report was 3.92-4.62s, with substantially higher inference efficiency than general-purpose models. Multi-reader evaluation demonstrated consistent improvements in QC performance (ΔF1-score: 0.031-0.257) and reduced QC time.

Conclusions:

The proposed RadQC-LLM ensures data security while enabling efficient inference and demonstrates robust performance comparable to or exceeding that of general-purpose LLMs, with strong potential for future clinical deployment. Clinical Trial: The study protocol was approved by the Medical Ethics Committee of Beijing Friendship Hospital, Capital Medical University (Approval No. 2025-P2-484-01). The requirement for informed consent was waived due to the retrospective nature of the study. Once approved by the Institutional Review Board/Ethics Committee of all participating hospitals, deidentified data can be made available through a secured online file transfer system for research purpose only.