Accepted for/Published in: JMIR Formative Research
Date Submitted: Apr 21, 2026
Date Accepted: Oct 21, 2026
Large Language Models for Structured Critical Appraisal of Randomized Controlled Trials: A Randomized Crossover Study Comparing ChatGPT-4o and Medical Students
ABSTRACT
Background:
Critical appraisal of medical literature is an essential skill for trainees but can be time-intensive and difficult to master. Large language models (LLM) such as ChatGPT have emerged as potential tools to support learners; however, their accuracy and reliability in structured appraisal tasks remain unclear.
Objective:
The objective of this study is to compare the performance of ChatGPT-4o with medical students in structured critical appraisal of randomized controlled trials (RCT), evaluating its potential role in medical education.
Methods:
In this prospective randomized crossover study, 12 RCTs published in The Laryngoscope in 2024 were identified and appraised using the Critical Appraisal Skills Programme (CASP) checklist. Two second-year medical students independently appraised each article with and without the assistance of ChatGPT-4o, separated by a 1-month washout period. The ChatGPT-4o LLM was also used to independently appraise each appraisal. Two otolaryngologists evaluated the randomized and anonymized appraisals using a standardized rubric assessing binary correctness, free-text accuracy, completeness, and overall quality. Medical students and evaluators also completed post-study questionnaires regarding artificial intelligence (AI) use and perceptions.
Results:
Binary correctness did not differ significantly amongst appraisals completed by students with AI assistance, students without AI assistance, and ChatGPT-4o alone. Students demonstrated significantly greater free-text accuracy than ChatGPT-4o alone (P<0.001), whereas no significant difference was observed between the two student conditions. Completeness and overall quality ratings were similar across all three groups. Physician evaluators correctly identified the presence of AI involvement in approximately 56% of appraisals. Overall interrater agreement of the physician reviewers was 91%. Medical students reported that ChatGPT-4o reduced the time required to complete appraisals and improved their understanding of the reviewed studies.
Conclusions:
ChatGPT-4o demonstrated performance comparable to medical students in binary appraisal questions, completeness, and overall quality but was less accurate in free-text critical appraisal responses. These findings support the cautious use of ChatGPT-4o as an adjunct educational tool for structured critical appraisal while emphasizing the continued importance of independent critical reasoning and human oversight. Given the small number of participating medical students, larger studies are needed to determine the generalizability of these findings and to evaluate AI-assisted critical appraisal across diverse study designs. Clinical Trial: Not applicable
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.