Previously submitted to: JMIR Formative Research (no longer under consideration since Aug 28, 2025)
Date Submitted: Mar 19, 2025
Open Peer Review Period: Mar 31, 2025 - May 26, 2025
(closed for review but you can still tweet)
NOTE: This is an unreviewed Preprint
Warning: This is a unreviewed preprint (What is a preprint?). Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn (a note "no longer under consideration" will appear above).
Peer review me: Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period (in this case, a "Peer Review Me" button to sign up as reviewer is displayed above). All preprints currently open for review are listed here. Outside of the formal open peer-review period we encourage you to tweet about the preprint.
Citation: Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
Final version: If our system detects a final peer-reviewed "version of record" (VoR) published in any journal, a link to that VoR will appear below. Readers are then encourage to cite the VoR instead of this preprint.
Settings: If you are the author, you can login and change the preprint display settings, but the preprint URL/DOI is supposed to be stable and citable, so it should not be removed once posted.
Submit: To post your own preprint, simply submit to any JMIR journal, and choose the appropriate settings to expose your submitted version as preprint.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Use of Artificial Intelligence for Evaluation of the Entrance Exam of the Brazilian Society of Shoulder and Elbow Surgery: Observational Study
ABSTRACT
Background:
Artificial intelligence (AI) has shown growing potential in medical education and assessment. However, its effectiveness in specialized exams, such as those administered by the Brazilian Society of Shoulder and Elbow Surgery (SBCOC), remains underexplored.
Objective:
This study aimed to compare the performance of multiple AI models against human candidates (national average) in the SBCOC entrance exams from 2021 to 2023, focusing on answer accuracy and reliability of cited sources.
Methods:
We evaluated five AI models—ChatGPT-4, ChatGPT-o1-pro, GEMINI (Google), Meta Llama 3.1, and a trained ChatGPT-4 version restricted to specific reference files—using the official SBCOC exams (50 multiple-choice questions per year). For each question, the AI provided a chosen answer (A/B/C/D) and a cited source. Image-based questions were handled by pasting the same image prompt into each AI interface. Outcomes included accuracy (comparison with the official key) and source reliability (scientific articles/books vs. websites). Statistical comparisons used chi-square and one-way ANOVA (significance level p<0.05) with post hoc tests, and 95% confidence intervals (CIs) were reported where applicable. This study was approved by the Research Ethics Committee of UNIFESP – Escola Paulista de Medicina, under CAAE number 81368024.9.0000.5505.
Results:
A total of 150 questions (50/year) were analyzed. Across the three years, ChatGPT-o1-pro mode achieved the highest accuracy (66% in 2021, 62% in 2022, 68% in 2023), significantly outperforming the average candidate scores by 14% (p=0.04), 20% (p=0.03), and 36% (p=0.029), respectively. GEMINI (40% in 2021; 28% in 2022; 38% in 2023) and Meta Llama 3.1 (32%, 44%, 50%) consistently underperformed compared to candidates (p>0.01). ChatGPT-4 (untrained) and trained ChatGPT-4 hovered near the passing threshold (>= 50%), displaying no statistically significant difference from candidate performance (p>0.05). Image-based questions showed lower AI consistency, highlighting limitations in visual interpretation.
Conclusions:
ChatGPT-o1-pro surpassed human performance over three consecutive exams. Both the untrained and trained ChatGPT-4 models performed comparably to the national average. While AI demonstrates considerable potential in multiple-choice medical exams, improved handling of image-based questions and more consistent reference verification remain crucial for broader clinical and educational applications. Clinical Trial: -
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.