JMIR Publications

ABSTRACT

Background:

The rapid acceleration of large language models (LLMs) has created new opportunities to expand the accessibility of mental health support. However, while nearly half of individuals report using a general LLM for mental health support in the past year, general artificial intelligence (AI) tools lack crucial safety guardrails, evidence-based practices, and compliance with medical regulations which may result in misinformation, perpetuating mental health stigma, and failing to escalate care in crisis situations. In contrast, Ebb, Headspace’s conversational AI tool, was purpose-built by clinical psychologists and research experts using motivational interviewing techniques for subclinical guidance, incorporating clinically-backed safety mechanisms.

Objective:

The purpose of this study was to: 1) understand Headspace members’ overall sentiment toward AI and expectations for an AI mental health tool; 2) evaluate the real-world use and engagement patterns of Ebb; and 3) understand how members perceive Ebb fitting into their broader mental health journey.

Methods:

This was a mixed methods study utilizing three data sources including Headspace members: 1) cross-sectional member survey (n=482 Ebb users) assessing demographics, general AI use, and AI attitudes (AIAS-4); 2) real-world engagement descriptive analysis using Headspace app data (n=393,969 Ebb users) analyzing session/message counts, retention, and conversation themes; and 3) a diary study (n=15 Headspace members) to qualitatively explore Ebb's role within members’ mental health journey. App engagement was compared between Ebb 1.0 and Ebb 2.0, where Ebb 2.0 featured enhanced LLM conversational prompts, comprehensive memory, woven content recommendations, and more robust safety detection.

Results:

While a majority of survey respondents used general AI tools (57.3%) and reported that they would use AI tools in the future, overall attitudes towards AI remained neutral (AIAS-4 mean=5.7, SD=2.2, range=1-10). Members viewed Ebb as a guide to navigate to mental health resources and Headspace content and a tool to provide in the moment support. Real-world Ebb use showed strong engagement across 393,969 Headspace members. The product evolution to Ebb 2.0 led to increased retention (50.8% completed two sessions within 7 days vs. 28.5% for Ebb 1.0) and higher positive conversation ratings (93.5% vs. 90.4%). Retained Ebb 2.0 users (2 or more sessions within 7 days) showed greater retention (6.1 sessions/user) compared to all Ebb 2.0 users (2.9 sessions/user) and Ebb 1.0 (2.4 sessions/user). Diary study results suggest that members imagined using Ebb when feeling anxiety/overwhelmed; during stressful moments; and during morning routines, commutes, or while winding down at night.

Conclusions:

Results emphasize the necessity of research-backed development for purpose-built AI mental health products with minimum viable safeguards including: 1) transparent consumer labeling of intended use, benefits, and limitations; 2) safety by design principles to monitor for overuse, detect risk, and flag needs for escalation; and 3) child and adolescent safeguards to account for developmental differences in users. Future research must connect these descriptive engagement findings with clinical outcomes and care adherence.