Previously submitted to: Journal of Medical Internet Research (no longer under consideration since Jul 06, 2023)
Date Submitted: Mar 13, 2023
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Large Language Models in Surgical Education: Do they Reach Human Level on Fundamentals of Robotic Surgery Test?
ABSTRACT
Background:
Large language models are capable of answering questions as if they were engaged in active conversation with users. However, currently, there are no data on whether their performances will remain static or vary over time when answering questions in the medical domain.
Objective:
The aim of the present study was to assess ChatGPT and InstructGPT on multiple trials on the Fundamentals of Robotic Surgery (FRS) test. Additionally, different releases of ChatGPT were compared to establish whether its performance improved after retraining.
Methods:
We tested the performance of ChatGPT and InstructGPT on the 44 multiple choice questions of FRS didactic test, for which a pass mark requires 35 correct answers (79.5%). Seven attempts were performed using ChatGPT on the January 30, 2023 release and seven with the February 13, 2023 version. Three trials were performed with InstructGPT.
Results:
ChatGPT achieved a mean score of 64.6% and 65.6% respectively for the first and second release, without any significant difference between the two (p = 0.32). The score ranged from 54.5% to 72.7% with both versions. On baseline it achieved 54.5% in both releases, higher than InstructGPT (50.0%). The highest rate of correct answers of ChatGPT was observed for questions on team training and communication (77.5% with both releases), followed by those on the introduction of the robotic system (67.5% and 62.7 % respectively for the first and second versions), psychomotor skills (64.3% and 57.1%), and naming correctly clinical steps of a procedure of robot-assisted surgery (53.8% and 65.9%).
Conclusions:
Even though ChatGPT did not pass FRS test in any of the 14 trials, the 72.7% score observed by the present study represents a remarkable result, taking into consideration the generic nature of ChatGPT as distinct from a domain specific LLM. This level represents the highest score by ChatGPT in a high-stake examination in medicine.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.