JMIR Publications

ABSTRACT

Background:

Emergency radiology necessitates highly accurate reporting under time constraints, yet increasing workloads raise the risk of errors. While large language models (LLMs) show potential for proofreading in general radiology, their performance in emergency settings and non-English contexts remains unclear.

Objective:

To evaluate the performance of a domain-optimized LLM, DeepSeek-R1, for identifying errors in Chinese emergency radiology reports, with comparison against assessments by board-certified radiologists.

Methods:

We compiled 7435 emergency reports (radiography, CT, MRI) collected from November 2024 to April 2025. In Stage 1, five LLMs were evaluated using 200 reports. The best model, DeepSeek-R1, proceeded to Stages 2 and 3, where zero-shot and few-shot learning were tested on a separate set (n = 100). Model performance was compared against 12 radiologists. Stage 4 validated real-world utility on 800 verified reports.

Results:

DeepSeek-R1 achieved higher error detection rate using few-shot compared to zero-shot settings (84.4% vs. 60.9%, P = 0.003), outperforming residents (84.4% vs. 51.6% and 53.1%, respectively, both P < 0.05) and matching senior radiologists and attendings (84.4% vs. 68.8-93.8%, P = 0.26-1.00). Compared to residents, it detected 100% of critical omissions and 92% of other errors (all P < 0.05). Reading time was faster than humans (92 vs. 109 seconds). In real-world validation, DeepSeek-R1 identified 117 true errors (PPV 56.5%).

Conclusions:

DeepSeek-R1 holds promise for improving quality control in emergency radiology reports. Its efficiency and accuracy support its use as an assistive tool in real-world settings.