JMIR Publications

ABSTRACT

Background:

Around 60-80% of primary care visits have a psychological stress component, but only 3% receive stress management advice. Given recent advances in natural language processing there is renewed interest in mental health chatbots. Conversational agents that can understand a user’s problems and deliver advice that mitigates the effects of daily stress could be an effective public health tool. However, such systems are complex to build and costly to develop.

Objective:

To address these challenges, our aim is to develop and evaluate a fully automated mobile suite of shallow chatbots—we call Popbots—that complement human support in an ecosystem of stress management support.

Methods:

After conducting an exploratory Wizard of Oz study (N=14) to evaluate the feasibility of a suite of multiple chatbots, we conducted an online study (N=47) to evaluate a prototype implementation. Each participant was randomly assigned to a different chatbot designed based on a proven cognitive or behavioral intervention method. To measure the effectiveness of chatbots, participant’s stress levels were determined using self-reported psychometric evaluations (e.g., online daily surveys, PHQ-4). Participants in these studies were recruited via email, were enrolled online, and some participated in follow-up interviews that were conducted in-person or online (as necessary).

Results:

Approximately 31 of 47 participants (69.5%) completed the main study. Findings suggest that users view conversations with our chatbots as helpful or at least neutral and come away with increasingly positive sentiment toward the use of chatbots for proactive stress management. Moreover, those users who used the system more (i.e., have greater than or equal to the median number of conversations) see a decrease in depression symptoms compared to those who used the system less based on a Wilcoxon Signed-rank test (W=91.50, Z=-2.54, p=.011, r=0.47). Follow-up interviews with a subset of participants indicated that half of common daily stressors could be discussed with chatbots, potentially reducing the burden on human coping resources.

Conclusions:

Our work suggests that suites of shallow chatbots may offer benefits for both users and designers. As a result, its contributions include: (i) the design and evaluation of a novel suite of shallow chatbots for daily stress management, (ii) a summary of benefits and challenges associated with random delivery of multiple conversational interventions, and (iii) design guidelines and directions for future research into similar systems, including authoring chatbot systems and AI-enabled recommendation algorithms.