JMIR Publications

ABSTRACT

Background:

There is a shortage of services available to address the growing demand for mental health support in Australia and worldwide. Digital interventions, including conversational agents, can overcome barriers to accessing mental health support. The recent advances in large language models (LLMs) have led to an improvement in the perceived human-like naturalness of chatbot conversations, but there is little research on the experience of chatbots to support mental health. Manage Your Life Online (MYLO) is a rule-based chatbot that was co-designed with young people and uses questions to help users explore their problems. In a case series conducted prior to the release of ChatGPT, users rated a new smartphone interface for MYLO as acceptable and there was a large effect size for reduction in problem-related distress.

Objective:

We aimed to evaluate an improved version of MYLO and compare the user experience of MYLO in this case-series to the previous version that was completed in November 2022.

Methods:

We replicated and extended the previous two-week case-series, conducted in September to November 2022, by testing four-week usage of MYLO with a larger sample between October and December 2023. We recruited 24 young people living in Western Australia who self-described as having lived experience of anxiety and/or depression. Participants had access to, and used, MYLO over a four-week period while completing online weekly surveys that included a range of health and psychological questionnaires. After the four-week testing phase, participants were invited to provide feedback on their experience of using MYLO through an interview or focus group discussion.

Results:

Thirteen of the initial 24 participants were retained throughout the study and took part in interviews. On average, participants had around four conversations with MYLO. They experienced both benefits and limitations of these conversations. They spoke about their recent experiences with ChatGPT (released in November 2022 after the previous case-series concluded) and other generative AI tools, stating that they had expected MYLO to possess similar functionality, which it did not. Nonetheless, we found moderate to large effect sizes for improvements in problem-related distress (d = -1.07), anxiety (d = -0.41) and psychiatric impairment (d = 0.60) and some evidence of reliable improvement in clinical outcomes.

Conclusions:

These findings have implications for mental health chatbots in the age of ChatGPT and highlight a need for researchers to engage with new technologies to improve user experience, while maintaining the necessary safety and ethical standards that can be a significant challenged for generative AI.