Accepted for/Published in: JMIR Medical Education
Date Submitted: Sep 2, 2026
Date Accepted: Oct 26, 2026
Medical Interview Performance Using an AI Standardized Patient Versus a Human Standardized Patient: A Randomized Crossover Trial
ABSTRACT
Background:
In Japan, standardized patients (SPs) have traditionally been recruited to teach medical interview competencies. Because of the time, cost, and effort associated with SPs, securing an adequate number of SPs remains challenging. Considering that the effectiveness of conversations with AI have advanced markedly, we considered assessing Human-SPs against AI-SPs powered by a large language model.
Objective:
This randomized single-center crossover trial aimed to compare medical interview training using large language model-based AI-SP with training using a Human-SP, and assessed the educational utility of AI-SPs through examinations of medical interviewing skills.
Methods:
Participants were medical students undertaking clinical clerkships at Chiba University Hospital. Participants were divided into two groups: one underwent a chat-based medical interview with an AI-SP (ChatGPT-4o), followed by an interview with the Human-SP; the other began with a Human-SP interview, followed by the AI-SP interview. A washout period of at least three days was set between conditions. For the AI-SP interviews, chat logs were retained, and a physician provided feedback after completion of the two cases. Human-SP interviews were video-recorded, and the physician portraying the SP provided feedback immediately after each interview. Medical interview performance was assessed using selected items from the mini-CEX, with ratings performed independently by two general medicine specialists. Participants completed a self-rated questionnaire (seven-point Likert-type scale) regarding learning satisfaction with the simulated interview, perceived rapport formation, and active engagement in history taking.
Results:
Forty-eight participants were analyzed. The mini-CEX-based total score was significantly higher for the Human-SP than for the AI-SP condition (23.9±1.7 vs. 22.7±1.7; mean difference, 1.2; t(47) = 3.7, P = .001). Scores for history taking and overall clinical competence were higher in the Human-SP condition, with nonsignificant differences for communication, diagnosis, management, or professionalism. Inter-rater reliability was ICC (2,1) = 0.76 (95% CI 0.66–0.83). No significant differences were found between conditions for learning satisfaction (Human-SP: 5.9±1.1; AI-SP: 6.0±0.8) or perceived rapport formation (Human-SP: 5.4±1.3; AI-SP: 5.3±1.3). Active engagement in history taking was significantly higher in the AI-SP condition (Human-SP: 5.8±1.1; AI-SP: 6.2±0.7; t(47) = 2.7, P = .009).
Conclusions:
Overall medical interview scores were significantly higher in the Human-SP than in the AI-SP condition, primarily owing to higher history-taking scores. One possible explanation is nonverbal cue availability in Human-SP encounters, which were limited in the text-based AI-SP condition. Although no significant differences between conditions were observed in most other assessment outcomes, learning satisfaction, or perceived rapport, self-rated active engagement in history taking was higher with the AI-SP condition. These findings suggest that AI-SPs may provide a psychologically accessible environment for active and repeated practice. While AI-SPs cannot replace Human-SPs, they may serve as complementary tools in medical interview education, particularly during early-stage training.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.