JMIR Publications

ABSTRACT

Background:

The rapid development of Large Language Models (LLMs) such as OpenAI's ChatGPT, has significantly impacted medical research and education. These models have shown potential in fields ranging from radiological imaging interpretation to medical licensing exam assistance. Recently, LLMs have been enhanced with image recognition capabilities.

Objective:

This study aims to critically examine the effectiveness of these LLMs in medical diagnostics and training by assessing their accuracy and utility in answering image-based questions from medical licensing examinations.

Methods:

The study analyzed 1070 image-based multiple-choice questions from the AMBOSS learning platform, divided into 605 in English and 465 in German. Customized prompts in both languages directed the models to interpret medical images and provide the most likely diagnosis. The student performance data was obtained from AMBOSS, including metrics such as the Student passed Mean and Majority Vote. Statistical analysis was conducted using Python, with key libraries for data manipulation and visualization.

Results:

GPT-4 1106 Vision Preview outperformed Bard Gemini Pro, correctly answering 56.9% of questions compared to Bard's 44.6%, a statistically significant difference (χ2₁ = 32.1, P<.001). However, GPT-4 1106 left 16.1% of questions unanswered, significantly higher than Bard's 4.1% (χ2₁ = 83.1, P<.001). When considering only answered questions, GPT-4 1106's accuracy increased to 67.8%, surpassing both Bard (46.5%, χ2₁ = 87.7, P<.001) and the student passed mean of 63% (χ2₁ = 4.8, P=.028). Language-specific analysis revealed both models performed better in German than English, with GPT-4 1106 showing greater accuracy in German (60.65% vs. 54.1%, χ2₁ = 4.4, P=.036) and Bard Gemini Pro exhibiting a similar trend (54.8% vs. 36.7%, χ2₁ = 34.3, P<.001). The student majority vote achieved 94.5% accuracy overall, significantly outperforming both AI models (vs. GPT-4 1106: χ2₁ = 408.5, P<.001; vs. Bard Gemini Pro: χ2₁ = 626.6, P<.001).

Conclusions:

Our study shows that GPT-4 1106 Vision Preview and Bard Gemini Pro have potential in medical visual question-answering tasks and to serve as a support for students. However, their performance varies depending on the language used, with a preference for German. They also have limitations in responding to non-English content. The high accuracy rates, particularly when compared to student responses, highlight the potential of these models in medical education, yet the need for further optimization and understanding of their limitations in diverse linguistic contexts remains critical.