JMIR Publications

ABSTRACT

Background:

The introduction of GPT-4 marked a significant milestone in AI, including medical applications, with an impressive 83.76% zero-shot accuracy on the USMLE. The introduction of its visual extension, GPT-4V, with multimodal capabilities opens up new possibilities in radiological analysis.

Objective:

This study evaluates the diagnostic capabilities of GPT-4 and GPT-4V in advanced radiological tasks, with the aim of assessing their utility in identifying pathological features in medical images.

Methods:

A total of 207 cases with 1312 images from the RSNA Case Collection were analyzed using both GPT-4 and GPT-4V. Each model underwent two tasks: diagnosing with differential diagnoses and answering multiple choice questions. For GPT-4V, the "chain-of-thought" prompting approach was used, which involves a step-by-step diagnostic reasoning process.

Results:

GPT-4V showed superior performance to GPT-4 on several measures. For primary diagnoses, GPT-4V achieved an accuracy of 27% (95% CI: 21%-34%) compared to 18% (95% CI: 12%-25%) for GPT-4. When differential diagnoses were included, GPT-4V's accuracy increased to 35% (95% CI: 29%-40%), surpassing GPT-4's 28% (95% CI: 22%-33%). In the second task, GPT-4V outperformed GPT-4 with an accuracy of 64% (95% CI: 59%-72%) versus 47% (95% CI: 42%-56%).

Conclusions:

Multimodal GPT-4V shows promise in handling complex radiology questions, suggesting its potential usefulness in medical image analysis. However, it is important to view this technology as a supportive tool alongside the expertise of medical professionals. This study highlights the need for more research with larger datasets to fully understand the capabilities and limitations of AI tools like GPT-4V in radiology.