JMIR Publications

ABSTRACT

Objective:

To explore the feasibility of using publicly available mainstream large language models (LLMs) to evaluate the consistency and diagnostic accuracy of standardized ultrasound imaging reports, using pathology as the reference standard. Materials and

Methods:

This retrospective study collected ultrasound imaging data of breast nodules with pathological diagnoses obtained at our hospital from June 2019 to June 2024. ChatGPT-4 was employed to evaluate the BI-RADS classification and benign or malignant nature of the nodules. The diagnostic performance of the LLM and the human-machine dialogue method (image-to-text–LLM) was assessed, including accuracy, sensitivity, specificity, and the area under the receiver operating characteristic curve (AUC).

Results:

A total of 671 patients (mean age 45.82 ± 9.20 years [SD]; age range 26–75 years) with 671 breast nodule ultrasound images (385 benign, 286 malignant) were included. ChatGPT-4 achieved an overall accuracy of 96.87% in identifying BI-RADS classifications, surpassing the performance of two junior radiologists. For image interpretation, ChatGPT-4 achieved an AUC of 0.82 (95% CI: 0.79–0.85), an accuracy of 80.63% (541 of 671 cases), a sensitivity of 90.56% (259 of 286 cases), and a specificity of 73.51% (283 of 385 cases). Its diagnostic performance was comparable to that of two senior radiologists and superior to two junior radiologists. When utilizing the image-to-text–LLM, diagnostic performance, including AUC, accuracy, sensitivity, and specificity, improved for all four radiologists. Conclusion: The application of LLMs (particularly the image-to-text–LLM) demonstrates potential value in diagnosing breast ultrasound imaging data.