JMIR Publications

ABSTRACT

Background:

University students experience elevated psychological distress with limited access to mental health services. While cognitive-behavioral therapy (CBT) demonstrates efficacy for anxiety and depression, treatment gaps persist due to access barriers and insufficient between-session support. Large language model (LLM) chatbots could improve and scale CBT delivery. However, the scientific evaluation of chatbot-enhanced protocols is just emerging.

Objective:

This pilot study assessed the feasibility, acceptability, and preliminary efficacy of integrating an LLM-based ChatBot with group Unified Protocol (UP) therapy for university students with subclinical anxiety and depression symptoms.

Methods:

A single-arm feasibility trial recruited university students aged ≥18 years with moderate subclinical symptoms (Social Phobia Inventory: 21-40, Patient Health Questionnaire-9: 5-14, or Generalized Anxiety Disorder-7: 5-14), excluding those with current psychiatric disorders, suicidal ideation, or psychotropic medication use. The intervention comprised 4 weekly group UP counseling sessions complemented by access to a Claude 3.7-Sonnet LLM ChatBot programmed with UP-based therapeutic prompts for between-session support. Primary feasibility outcomes included treatment adherence, chatbot engagement metrics, and system usability (System Usability Scale). Secondary outcomes assessed changes in generalized anxiety (GAD-7), social anxiety (SPIN), depression (PHQ-9), and well-being (Short Warwick Edinburgh Mental Well-Being Scale) using paired t-tests. Qualitative feedback was collected through focus group interviews and analyzed using thematic analysis.

Results:

Of 72 screened participants, 37 met eligibility criteria and 19 initiated treatment (mean age 22.06 years, SD 1.78; 70.6% female). Retention was high with 17 completers (10.5% dropout rate). Among completers, 94.1% (16/17) attended ≥3 group sessions. ChatBot engagement was substantial: participants were active on a median of 23 days during the 34-day study period and exchanged a median of 15 messages in total. System usability was rated as excellent (mean 84.94, SD 10.98 out of 100). Pre-to-post comparisons revealed significant improvements in generalized anxiety (mean change -3.00, SD 3.46; t16 = 3.01, P = .004; Cohen’s d = 0.71) and mental well-being (mean change +2.29, SD 3.65; t16 = -2.17, P = .023; Cohen’s d = 0.69). Social anxiety and depression showed non-significant trends toward improvement. Qualitative feedback highlighted the ChatBot's accessibility and non-judgmental support while noting limitations in personalization. No adverse events or inappropriate chatbot interactions occurred.

Conclusions:

Integrating an LLM chatbot with group UP therapy demonstrated high feasibility, acceptability, and preliminary efficacy signals for university students with subclinical symptoms. The hybrid intervention package achieved strong retention and engagement while maintaining safety. These findings support progression to a randomized controlled trial to definitively evaluate this technology-enhanced approach for expanding access to evidence-based mental health interventions.