JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) enable flexible conversational virtual patients and may facilitate scalable clinical interview training. However, evidence on their integration into routine, group-based undergraduate psychiatry teaching remains limited, particularly under real-world educational conditions involving multiple faculty members and student groups.

Objective:

This study aimed to develop a structured and scalable framework for creating LLM-based virtual psychiatric patients and to evaluate its feasibility, conversational performance, and perceived educational value through real-world implementation in routine undergraduate psychiatry teaching.

Methods:

A structured template for developing virtual psychiatric patients was created through clinical case review, multidisciplinary expert evaluation, and pilot testing, resulting in a 136-item framework. The methodology was subsequently implemented under real-world teaching conditions in mandatory psychiatry seminars involving 302 fourth-year medical students across four university hospitals. Five seminar topics were randomly assigned to the chatbot-based methodology and five to traditional teaching. In chatbot-based seminars, students collectively conducted clinical interviews with LLM-based virtual patients. Conversations were analyzed to characterize the clinical information explored and the consistency of chatbot responses. Students evaluated both methodologies using the validated Student Satisfaction with Teaching Methodology Questionnaire, with paired comparisons performed using Wilcoxon signed-rank tests. Faculty perceptions were assessed using an open-ended questionnaire.

Results:

Of 40 planned chatbot-based sessions, 27 (67.5%) were completed in full, with difficulties accessing the platform representing the main implementation barrier. Completed conversations contained a mean of 22.5 (SD 9.8) question-answer exchanges. Twenty-two questions addressed information not explicitly predefined in the template, and no chatbot response was judged inconsistent with the predefined clinical case. Sixty-eight students completed the satisfaction questionnaire. Compared with traditional teaching, chatbot-based seminars received higher scores for Perceived Learning Experience (median 0.769 vs 0.474; P<.001) and Self-Perceived Competence (median 0.667 vs 0.417; P<.001), and lower scores for Unpleasant Emotions (median 0.067 vs 0.133; P=.002). Of 29 faculty members who taught at least one chatbot-based seminar, 24 (82.8%) completed the faculty questionnaire; interactivity and participation were the most frequently reported advantages.

Conclusions:

A structured and scalable LLM-based framework enabled the development of flexible and clinically coherent virtual psychiatric patients and their real-world integration into routine group-based undergraduate teaching involving multiple faculty members. Students reported a more favorable learning experience, greater self-perceived competence, and fewer unpleasant emotions than with traditional seminars. However, the implementation barriers encountered under routine teaching conditions highlight the importance of technical accessibility and faculty preparation when translating LLM-based educational tools into practice. Clinical Trial: Not applicable. This study evaluated an educational intervention in medical students and faculty, with exclusively educational outcomes.