Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.
Who will be affected?
Readers: No access to all 28 journals. We recommend accessing our articles via PubMed Central
Authors: No access to the submission form or your user account.
Reviewers: No access to your user account. Please download manuscripts you are reviewing for offline reading before Wednesday, July 01, 2020 at 7:00 PM.
Editors: No access to your user account to assign reviewers or make decisions.
Copyeditors: No access to user account. Please download manuscripts you are copyediting before Wednesday, July 01, 2020 at 7:00 PM.
Capability of GPT-4V(ision) in Japanese National Medical Licensing Examination: Evaluation Study
Takahiro Nakao;
Soichiro Miki;
Yuta Nakamura;
Tomohiro Kikuchi;
Yukihiro Nomura;
Shouhei Hanaoka;
Takeharu Yoshikawa;
Osamu Abe
ABSTRACT
Background:
Previous research applying large language models (LLMs) to medicine was focused on text-based information. Recently, multimodal variants of LLMs acquired the capability of recognizing images.
Objective:
To evaluate the capability of GPT-4V, a recent multimodal LLM developed by OpenAI, in recognizing images in the medical field by testing its capability to answer questions in the 117th Japanese National Medical Licensing Examination.
Methods:
We focused on 108 questions that had one or more images as part of a question and presented GPT-4V with the same questions under two conditions: 1) with both the question text and associated image(s), and 2) with the question text only. We then compared the difference in accuracy between the two conditions using the exact McNemar’s test.
Results:
Among the 108 questions with images, GPT-4V's accuracy was 68% when presented with images and 72% when presented without images (P = .36).
Conclusions:
The additional information from the images did not significantly improve the performance of GPT-4V in the Japanese Medical Licensing Examination.
Citation
Please cite as:
Nakao T, Miki S, Nakamura Y, Kikuchi T, Nomura Y, Hanaoka S, Yoshikawa T, Abe O
Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: Evaluation Study