Previously submitted to: Journal of Medical Internet Research (no longer under consideration since Sep 16, 2024)
Date Submitted: Mar 3, 2024
Open Peer Review Period: Mar 7, 2024 - May 2, 2024
(closed for review but you can still tweet)
NOTE: This is an unreviewed Preprint
Warning: This is a unreviewed preprint (What is a preprint?). Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn (a note "no longer under consideration" will appear above).
Peer review me: Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period (in this case, a "Peer Review Me" button to sign up as reviewer is displayed above). All preprints currently open for review are listed here. Outside of the formal open peer-review period we encourage you to tweet about the preprint.
Citation: Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
Final version: If our system detects a final peer-reviewed "version of record" (VoR) published in any journal, a link to that VoR will appear below. Readers are then encourage to cite the VoR instead of this preprint.
Settings: If you are the author, you can login and change the preprint display settings, but the preprint URL/DOI is supposed to be stable and citable, so it should not be removed once posted.
Submit: To post your own preprint, simply submit to any JMIR journal, and choose the appropriate settings to expose your submitted version as preprint.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
The Use of Large Language Models Tuned with Socratic Methods on the Impact of Medical Students' Learning: A Randomised Controlled Trial
ABSTRACT
Background:
Large Language Models (LLM) are AI models that can generate conversational content based on a trained specified source of information (corpus).
Objective:
The aim is to use these corpus-trained LLMs to limit the content offered by LLM, then using prompt engineering to teach using Socratic methods.
Methods:
Two chatbots were created and deployed, powered by OpenAI’s GPT-3.5 model, with a medical-school textbook corpus. The first chatbot generates a brief summary and open-ended question. The second chatbot generates a case vignette from its pre-trained clinical cases, prompting users for a diagnosis. Both chatbots reply to the user’s response, commenting on the accuracy and asks further questions to encourage critical thinking. A randomised controlled trial was conducted on two groups comprising third year medical students. One group used both chatbots for 10 minutes while the other read the medical textbook. A 15-question test was administered to both groups before and after the intervention.
Results:
Forty students participated in the study. The average of the group before and after reading the textbook (n=20) are 3.9 +/- 1.0 and 7.6 +/- 1.5 respectively (p<0.001). The average of the group before and after using the bot (n=20) are 3.9 +/- 0.9 and 12.8 +/- 1.6 respectively (p<0.001). The respective increase in results was 3.7 and 8.9.
Conclusions:
Medical students’ learning showed a better performance using a LLM based chatbot compared to self-reading of medical information assessed using a standardised test. More studies are required to determine if LLM-based pedagogical methods are superior to standard education.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.