Accepted for/Published in: JMIR Medical Education
Date Submitted: Jul 25, 2025
Date Accepted: Nov 6, 2025
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Subject naïve students using AI LLM-GPTs can outperform students with subject knowledge, even while using internet search engines or e-books for problem-solving
ABSTRACT
Background:
In the modern era of education, technology has become an integral part of the learning process. The advent of advanced large language models, such as ChatGPT, along with the vast availability of information on online platforms like Google, enables educators to enhance the educational experience for students. This protocol aims to outline the effectiveness of AI LLM-GPTs in both naïve and learned students.
Objective:
To compare the scores in multiple choice questions when given the help of large AI natural language processing models (ChatGPT), internet search engines, and e-books in learned and naïve MBBS students
Methods:
A total of MCQ-based pharmacology questions were created and divided into four sets. These sets were given to the newly joined 2nd-year (naïve group) and the newly joined 3rd-year (learned group). 100 UG students from AMU, Aligarh, participated in this study. They were asked to answer these questions with the help of large AI natural language processing models (ChatGPT), internet search engines, and e-books. The marks were recorded, and the results were statistically compared between the naïve group and the learned group
Results:
In this study, learned students outperformed naïve students across all the learning modalities. However, both groups achieved their highest scores using AI-LLM (ChatGPT). Interestingly, naïve students using AI LLM-GPT scored higher than learned students using an internet search engine or e-books (p = 0.0107), emphasizing the superiority of the AI-LLMs in bridging knowledge gaps. Among all the methods, e-books yielded the lowest performance, especially for naïve users due to the time constraints. Effect sizes were highest for AI-assisted learning (partial η² = 0.328), emphasizing its strong impact on problem-solving performance
Conclusions:
In both groups, AI LLM-GPTs effectively helped the students in terms of problem-solving. The potential of AI-LLM in medical education is demonstrated by the ironic fact that the AI-LLM naïve group surpasses even the learnt group supplemented with an internet search engine in problem-solving abilities. To evaluate the study's generalisability and the long-term effects of AI-LLM on medical education and student-teacher relationships, more research is necessary. Clinical Trial: This cross-sectional study was conducted at a tertiary care teaching hospital. The Institutional Ethics Committee approved this study with the REF. NO: 1018/IEC/23/8/23
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.