JMIR Publications

ABSTRACT

Background:

The rapid acceleration of large language models (LLMs) has created new opportunities to expand the accessibility of mental health support. However, while nearly half of individuals report using a general LLM for mental health support in the past year, general artificial intelligence (AI) tools lack crucial safety guardrails, evidence-based practices, and compliance with medical regulations which may result in misinformation, perpetuating mental health stigma, and failing to escalate care in crisis situations. In contrast, Ebb, Headspace’s conversational AI tool (CAI tool), was purpose-built by clinical psychologists and research experts using motivational interviewing techniques for subclinical guidance, incorporating clinically-backed safety mechanisms.

Objective:

The purpose of this study was to: 1) understand Headspace members’ overall sentiment toward AI and expectations for a mental health CAI tool; 2) evaluate the real-world use and engagement patterns of Headspace’s CAI tool; and 3) understand how members perceive a CAI tool fitting into their broader mental health journey.

Methods:

This was a multiple method study utilizing three data sources including Headspace members: 1) cross-sectional member survey (n=482 Ebb users) assessing demographics, general AI use, and AI attitudes (AIAS-4); 2) real-world engagement descriptive analysis using Headspace app data (n=393,969 Ebb users) analyzing session/message counts, retention, and conversation themes; and 3) a diary study (n=15 Headspace members) to qualitatively explore the CAI tool’s role within members’ mental health journey. App engagement was compared between CAI Tool 1.0 and CAI Tool 2.0, where CAI Tool 2.0 featured enhanced LLM conversational prompts, comprehensive memory, woven content recommendations, and more robust safety detection.

Results:

While a majority of survey respondents used general AI tools (57.3%) and reported that they would use AI tools in the future, overall attitudes towards AI remained neutral (AIAS-4 mean=5.7, SD=2.2, range=1-10). Members viewed the CAI tool as a guide to navigate to mental health resources and Headspace content and a tool to provide in the moment support. Qualitative results suggest members used the CAI tool as an interactive self-reflection tool, to guide them toward Headspace content, and for in the moment support. Members emphasized the need for transparency in data safety and ethics, structure around clinical guidelines, and for the CAI tool to be a resource in addition to human-delivered mental health care not as a replacement. Real-world CAI tool use showed strong engagement across 393,969 Headspace members. The product evolution to CAI Tool 2.0 led to increased retention (50.8% completed two sessions within 7 days vs. 28.5% for CAI Tool 1.0) and higher positive conversation ratings (93.5% vs. 90.4%). Retained CAI Tool 2.0 users (2+ sessions within 7 days) showed greater retention (6.1 sessions/user) compared to all CAI Tool 2.0 users (2.9 sessions/user) and CAI Tool 1.0 (2.4 sessions/user). Diary study results suggest that members imagined using the CAI tool when feeling anxiety/overwhelmed; during stressful moments; and during morning routines, commutes, or while winding down at night.

Conclusions:

Results emphasize the necessity of research-backed development for purpose-built AI mental health products with minimum viable safeguards including: 1) transparent consumer labeling of intended use, benefits, and limitations; 2) safety by design principles to monitor for overuse, detect risk, and flag needs for escalation; and 3) child and adolescent safeguards to account for developmental differences in users. Future research must connect these descriptive engagement findings with clinical outcomes and care adherence.