JMIR Publications

ABSTRACT

Background:

Although large language models (LLMs) have demonstrated the ability to generate the impression section from radiology findings automatically, the incremental diagnostic value of clinical information for these models remains unclear.

Objective:

To evaluate the incremental diagnostic value of clinical information for LLMs and compare their performance with that of radiologists.

Methods:

This retrospective study included radiology reports from patients with histopathologically confirmed liver, lung, and breast diseases from two institutions between October 2021 and February 2025. We defined three progressive information input scenarios: (a) basic patient information and imaging findings; (b) Scenario a plus chief complaint or clinical history; and (c) Scenario b plus key laboratory results. Scenario-based data were input into three general-purpose LLMs (DeepSeek-R1, Gemini 2.5 Pro, and GPT-4o), generating 2709 entries. Diagnostic accuracy was assessed for both benign-malignant differentiation and disease diagnosis, with histopathology serving as the reference standard. Accuracy was compared among scenarios and against radiologist performance using McNemar tests, and P values were adjusted using the Holm–Bonferroni correction for multiple comparisons.

Results:

A total of 301 pathologically confirmed cases were included (mean age, 53.5 years ± 12.0 [SD]; 208 female). In the liver cohort, A numerical trend toward higher accuracy was observed in Scenario c compared with Scenario a across all three models (Scenario c range, 72.3%–76.2% vs Scenario a range, 64.4%–68.3%; all P > .05); however, these differences did not reach statistical significance after Holm–Bonferroni correction. Notably, the DeepSeek-R1 model in Scenario c achieved the highest diagnostic accuracy (76.2% [77 of 101]), with no evidence of a difference compared with radiologists (81.2% [82 of 101]; P = .18). In contrast, results in the lung and breast cohorts were more heterogeneous. In the lung cohort, GPT-4o achieved its highest accuracy in Scenario a for disease diagnosis (73.9%), which exceeded its performance in Scenario b (69.6%) and Scenario c (71.7%), suggesting that additional clinical information did not confer a consistent benefit. Gemini 2.5 Pro in Scenario b achieved the highest accuracy in this cohort (78.3% [72 of 92]); however, no statistically significant difference was found compared with radiologists (87.0% [80 of 92]; adjusted P = .25). In the breast cohort, the diagnosis accuracy was numerically highest in Scenario a, and it decreased numerically with the addition of laboratory tests, although no significant difference was found between Scenarios a and c (67.6% [73 of 108] vs 65.7% [71 of 108]; P = .77).

Conclusions:

While the addition of clinical information was associated with a numeric trend toward higher diagnostic accuracy overall, this trend was heterogeneous across models and disease types, and no statistically significant improvement was demonstrated after adjustment for multiple comparisons.