JMIR Publications

ABSTRACT

Background:

ChatGPT and language learning models have gained attention recently for their ability to answer questions on various examinations across various disciplines. The question of whether ChatGPT could be used to aid in medical education is yet to be answered, particularly in Ophthalmology.

Objective:

The aim of this study is to assess the ability of ChatGPT3.5 (GPT 3.5) and ChatGPT4.0 (GPT 4.0) to answer Ophthalmology related questions across different levels of Ophthalmology training.

Methods:

Questions for STEP 1(n=44), STEP 2 (n=60), and STEP 3 (n=28) were extracted from AMBOSS and 248 questions were extracted from the book “Ophthalmology Board Review Q&A” for the OKAP and Ophthalmology Board (OB) Written Qualifying Exam. Questions were prompted identically and put to GPT3.5 and GPT4.0.

Results:

GPT3.5 achieved a total of 55% correct, while GPT4.0 achieved a total of 70% correct. GPT3.5 performed poorer as examination levels advanced (p<0.001), while GPT4.0 achieved best on STEP 2 and STEP 3 , and worse on OB and STEP 1 (p<0.001). Correlation between GPT 3.5 answering correctly and human users answering correctly was r=0.21 (p=0.014) as compared to r= -0.31 for GPT4.0 (p<0.001). GPT3.5 performed similarly across difficulty levels, while GPT4.0 performed poorer as difficulty level increased. Both GPT models performed better on certain topics than others at significant levels.

Conclusions:

ChatGPT is far from being considered a part of mainstream medical education. Future models with higher accuracy are needed for the platform to be effective in medical education.