Accepted for/Published in: JMIR Medical Education
Date Submitted: Oct 28, 2023
Date Accepted: Mar 7, 2024
Using ChatGPT in Psychiatry to design script-concordance tests in undergraduate medical education: a mixed-method study.
ABSTRACT
Background:
Undergraduate medical studies represent a panoply of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently employed as a modality, followed by multiple choice and open-ended questions among other learning and teaching methods. As such, script-concordance tests (SCT) can be used to promote higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence-based systems like ChatGPT available to assist clinician-educators in creating instructional materials.
Objective:
The main objective of this project is to explore how SCT generated by ChatGPT compared to SCT produced by clinical experts on three major elements: the scenario (stem), the clinical questions and the expert opinion.
Methods:
This mixed-method study evaluated three ChatGPT-generated SCT with three expert-created SCT using a predefined framework. Clinician-teachers as well as resident doctors in psychiatry involved in the undergraduate medical education in the area of Quebec (Canada) evaluated online the six SCT over three criteria: scenario, questions and expert opinion. They were also asked to describe the strengths and weaknesses of the SCT.
Results:
A total of 102 respondents assessed the SCT. Findings demonstrated no significant distinctions between the two SCT types concerning scenarios (P=0.68, clinical questions (P=0.27), and expert opinions, as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT-generated and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with DSM-5 criteria, albeit with a tendency towards caricatured scenarios and simplistic content.
Conclusions:
This study is the first to concentrate on the design of SCT supported by artificial intelligence in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials and further validation is essential to ensure educational efficacy and accuracy.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.