JMIR Publications

ABSTRACT

Background:

There is a shortage of services available to address the growing demand for mental health support in Australia and worldwide. Digital interventions, including conversational agents, can overcome barriers to accessing mental health support. The recent advances in large language models (LLMs) have led to an improvement in the perceived human-like naturalness of chatbot conversations, but there is little research on the experience and efficacy of chatbots to support mental health. Manage Your Life Online (MYLO) is a rule-based chatbot that was co-designed with young people that uses questions to help users explore their problems. In a case series conducted prior to the release of ChatGPT, users rated a new smartphone interface for MYLO acceptable and results demonstrated a reduction in problem-related distress.

Objective:

To assess a later version of MYLO’s impact on target outcomes over a four-week period in a sample of young people with lived experience of anxiety and/or depression. We hypothesized that these results would replicate the previous case-series for problem-related distress, anxiety, psychiatric impairment and goal conflict reorganization. We also anticipated the longer usage time would lead to improvements on general health, depression and self-efficacy. We also aimed to compare the user experience of MYLO in this case-series to the previous version that was completed in November 2022.

Methods:

We replicated and extended the previous two-week case-series, conducted in September to November 2022, by testing four-week usage of MYLO, in October to December 2023, with a larger sample. To do this we recruited 24 young people living in Western Australia who self-described as having lived experience of anxiety and/or depression. Participants had access to and used MYLO over a four-week period while completing online weekly surveys that included a range of health and psychological questionnaires. After the four-week testing phase participants were invited to attend either an interview or focus group to provide feedback on their experience of using MYLO.

Results:

We found improvements in problem-related distress (d = -1.07), anxiety (d = -0.41) and psychiatric impairment (d = 0.60) and some evidence of reliable improvement in clinical outcomes. However, satisfaction with MYLO conversations was rated more poorly compared to the previous study. In qualitative interviews, participants spoke about their experiences with ChatGPT (released in November 2022 after the previous case-series concluded) and other generative AI tools, stating that they had expected MYLO to possess similar functionality.

Conclusions:

These findings have implications for mental health chatbots in the age of ChatGPT and highlight a need for researchers to engage with new technologies to improve user experience, while maintaining necessary safety and ethical standards.