JMIR Publications

ABSTRACT

Background:

Motivational interviewing (MI) is an effective approach for supporting health behaviour change, but face-to-face delivery is resource-intensive and difficult to scale. Rule-based Conversational Agents (CA) can improve access, yet their scripted interactions and limited language flexibility constrain MI delivery. This study evaluates the feasibility of using Large Language Models (LLMs) to deliver text-based MI coaching sessions.

Objective:

To describe the development of an LLM-based MI CA “Aimi”, and to evaluate its performance alongside a human coach delivering text-based MI and a rule-based MI-inspired CA in a physical activity behaviour change context.

Methods:

We developed Aimi using structured LLM-workflows that aim to enhance MI fidelity. We conducted a within-subjects study, where 18 adults interacted with (i) Aimi, (ii) an MI-trained human coach and (iii) a rule-based CA during live text-based role-play coaching sessions, in a randomised order. Transcripts were independently evaluated by an MI expert using the Manual for the Motivational Interviewing Skill Code Version 2.0 (MISC-2) to assess MI competency and fidelity. Participants completed a user experience questionnaire to provide general feedback and to assess session alliance, dialogue relevance, empathy, engagement, linguistic quality, and perceived motivation to change. Qualitative feedback was thematically summarized and categorized under strengths and weaknesses for each approach.

Results:

Across MISC-2 summary metrics, Aimi achieved higher fidelity scores than the trained human coach and rule-based CA, showing higher reflection-to-question ratios, more complex reflections, and greater elicitation of client change talk (91% vs. 71%). User experience ratings showed no significant differences across conditions. Qualitative analysis revealed distinct strengths and limitations across the coaching interactions: participants describing Aimi’s interactions as personalized, fluid, and adaptive, though sometimes overly reflective and lengthy; the human coach was viewed as empathetic and supportive but slow to respond; and the rule-based coach was viewed as efficient and structured, yet limited in depth and personalisation.

Conclusions:

This study demonstrates that LLM-based CAs can deliver MI with fidelity comparable to trained coaches while more reliably eliciting client change talk than trained humans or rule-based systems. The perceived robotic quality and response style represent important areas for future refinement. LLM-based CAs orchestrated through structured workflows offer a scalable pathway to deliver MI for health behaviour change.