JMIR Publications

ABSTRACT

Background:

The Motivational Interviewing (MI) approach has been shown to help move ambivalent people who smoke toward the decision to quit smoking. There have been several attempts to broaden access to such therapy through text-based chatbots. These typically employ scripted responses to client statements, but such non-specific responses have been shown to reduce effectiveness. Recent advances in Natural Language Processing provide a new way to create responses that are specific to a client’s statements, using a generative language model.

Objective:

To design, evolve and measure the effectiveness of a chatbot system that guides ambivalent people who smoke towards the decision to quit smoking with MI-style generative reflections.

Methods:

An interdisciplinary collaboration among MI-expert clinicians, computer engineers and social scientists designed a chatbot system that engages with smokers and employs elements of MI. A total of 349 ambivalent smokers were recruited online and interacted with one of four different versions of the chatbot. Participants’ readiness to quit was measured prior to the conversation and one week later using an 11-point scale, which measures three attributes related to smoking cessation: readiness, confidence, and importance. The number of quit attempts made in the week prior to the conversation and the week following was surveyed, and participants also rated the perceived empathy of the chatbot. The main body of the conversation consists of five scripted questions, responses from participants and then (for three of the four versions) generated reflections. To generate context-specific, MI-consistent reflections, a pre-trained Transformer-based neural network was fine-tuned on examples of high-quality reflections. The four versions of the chatbot are one in which only questions were asked without generated reflections, one with an initial version of the reflection generator, another with an improved generator, and one where the overall interaction was extended.

Results:

All four versions of the chatbot were associated with a significant positive increase in participants’ confidence to quit one week later compared to prior to the conversation. The increase in average confidence using the non-generative version was 1.0 (P=.001), whereas for the three generative versions, increases ranged from 1.2 - 1.3 (P < .001). The extended conversation with improved generative reflections was the only version associated with a significant increase in average importance (0.7, P <.001) and readiness (0.4, P=.0098). The enhanced reflection and extended conversations exhibited significantly better perceived empathy than the non-generative conversation (P =.02 and P=.004, respectively). The number of quit attempts did not significantly change between the week before and the week after the conversation across all four conversations, nor did the number of participants whose ambivalence appeared to resolve.

Conclusions:

The results suggest that generative reflections increase the impact of a conversation on readiness to quit one week later, although a significant portion of the impact seen so far can be achieved by only asking questions and not producing reflections. These results support further evolution of the chatbot conversation, and can serve as a basis for comparison against more advanced versions.