JMIR Publications

ABSTRACT

Background:

Training mental health clinicians to conduct standardized clinical assessments is challenging due to a lack of scalable, realistic practice opportunities. Traditional methods often fail to prepare trainees for the variability and complexity of real-world patient interactions, potentially impacting data quality in clinical trials. This paper introduces a novel approach to address this training gap using large language model (LLM)-based interview simulations.

Objective:

This study aims to develop and validate a voice-enabled virtual patient simulation system as a proof-of-concept. We describe the development of the system and evaluate whether it can generate virtual patients that (1) accurately adhere to pre-defined clinical profiles, (2) maintain a coherent and consistent narrative, and (3) produce dialogue that is perceived as realistic.

Methods:

We implemented a system that uses a LLM to simulate patients with specified symptom profiles, demographic backgrounds, and distinct communication styles. The system’s performance was analyzed through a mixed-methods evaluation, which included a formal assessment by 5 experienced clinical raters who conducted simulated structured MADRS interviews on 4 virtual patient personas, scored them on the scale and provided qualitative feedback on the system’s clinical plausibility, narrative cohesion and dialogue realism.

Results:

Across a total 20 interviews, the virtual patients demonstrated strong adherence to their configured clinical profiles, with human raters scoring the patients with high accuracy against their predefined score configurations. The mean item difference between MADRS rater scores and configured scores was 0.52 (SD = 0.75); Inter-rater reliability across items was 0.90 (95% CI =0.68-0.99). Expert raters consistently gave average ratings of “Agree” to “Strongly Agree” when asked to evaluate the qualitative realism and cohesion of the virtual patients.

Conclusions:

LLM-powered virtual patient simulations offer a promising, scalable tool for training clinicians in standardized clinical assessment. This pilot study provides initial evidence for the system’s ability to produce high-fidelity, clinically relevant practice scenarios.