JMIR Publications

ABSTRACT

Background:

Artificial intelligence (AI) and deep learning are revolutionizing the field of medicine, including gastroenterology. Notable among these advancements are large language models (LLMs) like ChatGPT and Bard. While their effectiveness in the United States Medical Licensing Examination has been evaluated, their performance in non-English tests, such as the Korean Medical Licensing Examination (KMLE), remains underexplored.

Objective:

This study examined the performance of LLMs in the KMLE and assessed the explanation quality for gastroenterology questions.

Methods:

We analyzed the performance of three LLMs, ChatGPT (GPT-3.5), GPT-4, and Bard in the official KMLE questions from 2021 and 2022 by focusing on their accuracy and consistency. The questions were categorized based on their subject and the inclusion of images. The quality of explanations provided by the LLMs for gastroenterology questions was assessed using the Global Quality Score (GQS).

Results:

In KMLE 2021 and 2022, GPT-4 achieved accuracy rates of 72.1% and 62.5%, respectively, with consistency of approximately 60%. These rates were significantly higher than those of ChatGPT and Bard. The higher accuracy was maintained across most subjects, as well as in questions with images. For gastroenterology questions, GPT-4 achieved accuracy rates of 55.8% and 63.9% in KMLE 2021 and 2022, respectively. The GQS of GPT-4 was significantly higher than those of other models, even in cases of incorrect responses.

Conclusions:

Unlike ChatGPT and Bard, GPT-4 achieved passing grades for the KMLE 2021 and 2022, and demonstrated high-quality explanations in gastroenterology. Further research on multimodal models and the use of prompt engineering is warranted.