JMIR Publications

ABSTRACT

Background:

Large Language Models (LLMs) have demonstrated significant potential in diverse domains, including medicine. Nonetheless, there is a scarcity of studies examining their performance in medical examinations, especially those conducted in languages other than English, and in direct comparison with medical students. Analyzing the performance of LLMs in state medical exams can provide insights into their capabilities and limitations and evaluate their potential role in medical education and examination preparation.

Objective:

To assess and compare the performance of three LLMs, GPT-4, Bing, and GPT-3.5-Turbo, in the German Medical State Exams of 2022 and evaluate their performance relative to medical students.

Methods:

The LLMs were assessed on a total of 630 questions from the Spring and Fall German Medical State Exams of 2022. The performance was evaluated with and without media-related questions. Statistical analyses included one-way ANOVA and independent t-tests for pairwise comparisons. The relative strength of the LLMs in comparison to the students was also evaluated.

Results:

GPT-4 achieved the highest overall performance, correctly answering 88.1% of questions, closely followed by Bing at 86.0%, and GPT-3.5-Turbo at 65.7%. The students had an average correct answer rate of 74.6%. Both GPT-4 and Bing significantly outperformed the students in both exams. When media questions were excluded, Bing achieved the highest performance with 90.7%, closely followed by GPT-4 with 90.4%, while GPT-3.5-Turbo lagged at 68.2%. There was a significant decline in the performance of GPT-4 and Bing in the Fall exam, which was attributed to a higher proportion of media-related questions and a potential increase in question difficulty.

Conclusions:

Large language models, particularly GPT-4 and Bing, demonstrate potential as valuable tools in medical education and for pre-testing examination questions. Their high performance, even in comparison to medical students, indicates promising avenues for further development and integration into the educational and clinical landscape.