JMIR Publications

ABSTRACT

Background:

Generative artificial intelligence (AI) conversational agents are increasingly integrated within digital mental health interventions (DMHIs). However, empirical data on real-world engagement, usage patterns, and satisfaction with generative AI conversational agents is limited.

Objective:

The purpose of this study was to examine real-world engagement among users who interacted with the generative AI conversational agent within Mental, a DMHI designed to support mental health. We aimed to: 1) characterize users engaging with Mental’s AI conversational agent, 2) examine real-world usage patterns of the AI conversational agent, 3) examine satisfaction and user feedback following sessions with the AI conversational agent, and 4) explore preliminary drivers of engagement with the AI conversational agent.

Methods:

This retrospective study analyzed naturalistic user data from 3,512 paid subscribers who engaged with Mental’s AI conversational agent between October 2024 and January 2026. At onboarding, users reported sex, current mindset, emotional distress level, desire for greater discipline, and primary stressors. After each session, users rated their satisfaction with the AI interaction. Engagement metrics (e.g., session frequency, timing, duration, and return rates) were derived from timestamped app usage data. Descriptive statistics were used to characterize users and usage patterns. Session satisfaction was compared across temporal engagement variables using ANOVAs and independent samples t-tests. A mixed-effects logistic regression was used to explore preliminary predictors of user return for a subsequent session.

Results:

Users were predominantly male (2,797/3,512; 79.6%) and reported moderate-to-high emotional distress at baseline (29,814/47,721; 62.5%) and were commonly completed in the evening (14,071/47,721; 29.5%). Mean session satisfaction was high (mean = 4.5/5), and did not vary by time of day or day of week. Users most frequently characterized sessions as “Insightful” (1433/2,089; 68.7%), “Good Advice” (1315/2,089; 63.1%), and “Felt Seen” (1247/2,089; 59.7%). The session-to-session return rate was 92.7% (i.e., proportion of sessions followed by a subsequent session). Session satisfaction was a significant predictor of return rate (odds ratio [OR] = 1.34, 95% CI [1.13, 1.59], p <.001), and this relationship did not differ by user mindset.

Conclusions:

Users engaged with an AI conversational agent within the Mental app outside of traditional care hours and reported high satisfaction and return rates. Generative AI conversational agents may support real-world engagement, including among individuals who face barriers to traditional mental health services. Future research should determine whether these engagement patterns translate into clinically meaningful outcomes.