JMIR Publications

ABSTRACT

Background:

Motivational Interviewing (MI) is widely used in preventive interventions, yet coding MI techniques and monitoring intervention adherence remain resource-intensive due to the reliance on manual transcription and expert review. Large language models (LLMs) offer a promising approach to automate these tasks, but their agreement with human coders in the context of prevention interventions has not been established.

Objective:

This study evaluated the agreement between an Artificial Intelligence (AI)-based coder (OpenAI's gpt-4.1) and trained human coders on two tasks: (a) identification of MI techniques (e.g., open questions, affirmations, giving of information) at the facilitator-message level, and (b) completing a 21-item checklist of implementation adherence for a brief MI-based preventive intervention for adolescent substance use.

Methods:

Two certified MI facilitators independently coded 72 facilitator messages from two MI session of MI techniques and completed a 21-item MI implementation adherence checklist. The AI-based coder classified the same facilitator messages and adherence checklist using a structured prompt derived from the MI facilitator coding manual. Intercoder agreement in use of MI techniques and implementation adherence was assessed using Cohen's kappa, Fleiss' kappa, and Cochran's Q tests.

Results:

For use of MI techniques, the AI coder demonstrated moderate to substantial agreement with human coders across most techniques, including open questions (k = .66-.69), affirmations (k = .66-.77), and giving of information (k = .91). No statistically significant differences in percentages of MI technique use were observed among the three coders. For MI implementation adherence, overall agreement was moderate (Fleiss' k = .487).

Conclusions:

These findings provide support for the feasibility of using LLMs to recognize MI techniques and assess implementation adherence. Results support a human-AI collaborative model in which the AI coder “pre-codes” facilitator messages and flags sessions for expert review, while human coders retain responsibility for higher-inference judgments. Future research should compare different LLMs and evaluate whether AI-assisted coding improves the scalability of routine implementation monitoring.