Previously submitted to: JMIR Medical Education (no longer under consideration since Dec 18, 2024)
Date Submitted: Sep 24, 2024
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Evaluating the Performance of Large Language Models (ChatGPT 4, Gemini 1.5 Pro, Command R+, and Llama 3 70B) on the Turkish Medical Specialty Training Entrance Exam
ABSTRACT
This study aims to evaluate the performance of advanced large language models (LLMs) - OpenAI-ChatGPT 4, Google AI-Gemini 1.5 Pro, Cohere-Command R+ and Meta AI-Llama 3 70B on questions within the scope of Basic Medical Sciences and Clinical Medical Sciences exams of the Turkish Medical Specialty Training Entrance Exam (2021, 1st semester) and to analyze the answers for user interpretability in languages other than English. This study used ChatGPT 4, Gemini 1.5 Pro, Command R+, and Llama 3 70B to test artificial intelligence models' medical knowledge and ethical evaluation capabilities. The research was conducted on the questions of Turkish Medical Specialty Training Entrance Exam held on March 21, 2021. Turkish Medical Specialty Training Entrance Exam is organized by the Student Selection and Placement Centre with 240 questions. These questions are divided into two main categories: questions that test the basic medical knowledge and ethics required of a doctor and clinical scenarios covering various diseases. Regarding Model Comparison, ChatGPT 4 was the best-performing model in most domains, with an overall accuracy of 88.75%. Its strengths lie in its ability to accurately answer basic and clinical medical science questions. Llama 3 70B followed closely behind with an overall accuracy of 79.17%. Although it could not quite match the performance of ChatGPT 4, it still showed strong knowledge consistency across various fields. Gemini 1.5 Pro and Command R+ lagged with overall accuracy rates of 78.13% and 50%, respectively. Furthermore, GPT-4 and Llama 3 70B achieved satisfactory results on the Turkish Medical Specialty Training Entrance Exam, demonstrating that it is a safe source for basic medical sciences and clinical medical sciences in languages other than English. ChatGPT 4 is currently the most suitable model for answering medical science questions in various domains. Gemini 1.5 Pro and Command R+ show potential but need significant improvement to compete with the best-performing models. LLMS could be a helpful resource for medical education and clinical support in areas where English is not the primary language, like Turkey.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.