JMIR Publications

ABSTRACT

Background:

With the rise of computer science and artificial intelligence, analyzing large datasets promises enormous potential in gaining insights for developing and improving evidence-based health interventions. One such intervention is the counseling strategy Motivational Interviewing (MI), which has been found effective in improving a wide range of health-related behaviors. Despite the simplicity of its principles, MI can be a challenging skill to learn and requires expertise to apply effectively.

Objective:

This study aims to investigate the performance of AI models in classifying Motivational Interviewing behavior and the feasibility of using these models in online helplines for mental health as an automated support tool for counselors in clinical practice.

Methods:

We used a coded dataset of 253 MI counseling chat sessions from the 113 Suicide Prevention helpline. With 23,982 messages coded with the MI-SCOPE codebook, we trained and evaluated four machine learning (ML) models and one deep learning model to classify client- and counselor MI behavior based on language use.

Results:

The deep learning model BERTje outperformed all ML models, accurately predicting counselor behavior (accuracy = 0.72, AUC = 0.95, Cohen’s kappa = 0.69). It differentiated MI congruent- and incongruent counselor behavior (AUC = 0.92, kappa = 0.65) and evocative and non-evocative language (AUC = 0.92, kappa = 0.66). For client behavior, the model achieved an accuracy of 0.70 (AUC = 0.89, kappa = 0.55). The model’s interpretable predictions discerned client change- and sustain talk, counselor affirmations, and reflection types, facilitating valuable counselor feedback.

Conclusions:

The results of this study demonstrate that AI techniques can accurately classify MI behavior, indicating their potential as a valuable tool for enhancing MI proficiency in online helplines for mental health. Provided that the dataset size is sufficiently large with enough training samples for each behavioral code, these methods can be trained and applied to other domains and languages, offering a scalable and cost-effective way to evaluate MI adherence, speed up behavioral coding, and provide therapists with personalized, quick, and objective feedback. Clinical Trial: This study is a secondary analysis of an existing unregistered clinical trial and primary analyses have been published. Janssen W, van Raak J, van der Lucht Y, van Ballegooijen W, Mérelle S. Can Outcomes of a Chat-Based Suicide Prevention Helpline Be Improved by Training Counselors in Motivational Interviewing? A Non-randomized Controlled Trial. Front Digit Health 2022;4:871841. Published 2022 Jun 21. doi:10.3389/fdgth.2022.871841 The ethics review committee of the VU University Medical Centre in Amsterdam reviewed and approved this study (2020.105).