Accepted for/Published in: JMIR Medical Education
Date Submitted: Apr 4, 2024
Open Peer Review Period: Apr 5, 2024 - May 31, 2024
Date Accepted: Jun 27, 2024
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Educational Utility of Clinical Vignettes Generated in Japanese by ChatGPT-4: A Mixed Methods Study
ABSTRACT
Background:
Evaluating the accuracy and educational utility of AI-generated medical cases, especially those produced by large language models (LLMs) models like ChatGPT-4, developed by OpenAI, is crucial yet underexplored.
Objective:
To assess the educational utility of ChatGPT-4-generated medical cases and their applicability in educational settings.
Methods:
Utilizing a Convergent Mixed Methods Design, this study conducted a web-based survey from January 8 to January 28, 2024, to evaluate 18 medical cases generated by ChatGPT-4 in Japanese. Feedback was solicited from a panel of physicians specializing in general internal medicine and/or general medicine (GIM/GM) and experienced in medical education. Chi-square and Mann-Whitney U tests were performed to identify differences among cases and linear regression was used to examine trends associated with physicians' experience. Thematic analysis of qualitative feedback aimed to identify areas for improvement and confirm the educational utility of the cases.
Results:
Out of 73 invited participants, 71 (97.3%) responded. The respondents, primarily male (64 of 71), spanned a broad range of practice years (1976-2017) and represented diverse hospital sizes throughout Japan. The majority deemed the Information Quality (IQ) (0.77) and Information Accuracy (IA) (0.68) as satisfactory, with these responses being based on binary data. The average scores assigned were 3.55 for Educational Usefulness (EU), 3.70 for Clinical Match (CM), 3.49 for Terminology Accuracy (TA), and 2.34 for Diagnostic Difficulty (DD), based on a 5-point Likert scale. Statistical analysis revealed significant variability in content quality and relevance across the cases (p<.001 after Bonferroni correction). Participants suggested improvements in generating physical findings, using natural language, and enhancing medical terminology accuracy. Thematic analysis highlighted the need for clearer documentation, clinical information consistency, content relevance, and patient-centered case presentations.
Conclusions:
ChatGPT-4-generated medical cases written in Japanese possess considerable potential as resources in medical education, with recognized adequacy in quality and accuracy. Nevertheless, there is a notable need for enhancements in the precision and realism of case details. This study emphasizes ChatGPT-4's value as an adjunctive educational tool in medical field, requiring expert oversight for optimal application.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.