JMIR Publications

ABSTRACT

Background:

Online mental health support plays a crucial role in promoting the mental health of college students. However, non-specialist online mental health helpers often lack professional training in empathy. Although AI-assisted simulation training has shown promise, existing studies have rarely provided integrated evidence on the reliability of automated empathy assessment, the short-term effectiveness of training, and the transferability of trained skills to authentic peer-support settings.

Objective:

The goal of this study was to develop a large language model-based Patient Bot training system based on real-world cases and comprehensively examine its reliability, effectiveness, and feasibility in training non-professional helpers, providing a foundation for further applications of AI-assisted peer training.

Methods:

We developed Patient Bot from a Chinese-language text corpus tailored to university mental health support contexts, including depressed university students’ text-based peer support dialogues. GLM-4-9B was fine-tuned on this corpus to enable Patient Bot to portray distressed university students in multi-turn online conversations. The system also incorporated an EPITOME-based automated empathy scoring module that evaluated helpers’ responses across three dimensions: emotional response, interpretation, and exploration, thereby providing structured feedback for empathy training. To examine whether Patient Bot could improve peer helpers’ empathic competence and whether trained skills could transfer to authentic helping contexts, two linked studies were conducted. In Study A, 66 university-based peer helpers were randomly assigned to either an AI-assisted training group or a standard empathy training group. The AI-assisted group completed five consecutive days of 10-minute text-based conversations with Patient Bot, and changes in empathy, supportive communication, and session management self-efficacy were compared between groups. In Study B, peer helpers from the two training conditions were paired with support seekers for 30-minute one-to-one online text-based support sessions. The two groups were compared in terms of help seeker–reported psychological outcomes, perceived empathy, relationship quality, and session experience.

Results:

1) the experimental group showed significant improvements in empathy ability (F (1, 61) = 5.841, P = .019 < 0.05, partial η2 = 0.087), communication skills (F (1, 61) = 18.128, P < 0.01, partial η2 = 0.229), self-efficacy (F (1, 61) = 8.697, P = .005 < 0.01, partial η2 = 0.125), and AI empathy ratings (F (4, 128) = 22.56, P < 0.001, partial η2 = 0.410); 2) among the three dimensions of empathy ratings—emotional response, explanation, and exploration—the explanation dimension did not show significant improvement (F (4, 128) = 2.143, P = .08 > 0.05, partial η2 = 0.181); 3) when grouping participants by initial empathy levels, those in the low-empathy group (F = 9.926, P < 0.01, partial η2 = 0.586) exhibited a more pronounced increase in empathy scores compared to the high-empathy group (F = 2.579, P = .06 > 0.05, partial η2 = 0.269). 4) Participants rated the Patient Bot training system positively in terms of satisfaction.

Conclusions:

This study provides that the Patient Bot training system effectively enhances mental health helpers’ empathic ability in a short time frame. Furthermore, the training positively impacted participants' overall communication skills and self-efficacy in helping situations. This AI-based training approach offers a feasible and efficient method to accelerate the development of psychological helping skills.