JMIR Publications

ABSTRACT

Background:

The increasing deployment of Conversational Artificial Intelligence (CAI) in mental health interventions necessitates an evaluation of their efficacy in rectifying cognitive biases and recognizing affect in human-AI interactions. These biases, including theory of mind and autonomy biases, can exacerbate mental health conditions such as depression and anxiety.

Objective:

This study aimed to assess the effectiveness of therapeutic chatbots (Wysa, Youper) versus general-purpose language models (GPT-3.5, GPT-4, Gemini Pro) in identifying and rectifying cognitive biases and recognizing affect in user interactions.

Methods:

The study employed virtual case scenarios simulating typical user-bot interactions. Cognitive biases assessed included theory of mind biases (anthropomorphism, overtrust, attribution) and autonomy biases (illusion of control, fundamental attribution error, just-world hypothesis). Responses were evaluated on accuracy, therapeutic quality, and adherence to Cognitive Behavioral Therapy (CBT) principles, using an ordinal scale. The evaluation involved double review by cognitive scientists and a clinical psychologist.

Results:

The study revealed that general-purpose chatbots outperformed therapeutic chatbots in rectifying cognitive biases, particularly in overtrust bias, fundamental attribution error, and just-world hypothesis. GPT-4 achieved the highest scores across all biases, while therapeutic bots like Wysa scored the lowest. Affect recognition showed similar trends, with general-purpose bots outperforming therapeutic bots in four out of six biases.

Conclusions:

General-purpose chatbots demonstrated superior capabilities in cognitive bias rectification and affect recognition compared to therapeutic bots. However, the results highlight the need for further refinement of therapeutic chatbots to enhance their efficacy and ensure safe, effective use in digital mental health interventions. Future research should focus on improving affective response and addressing ethical considerations in AI-based therapy.