JMIR Publications

ABSTRACT

Background:

Opioid use disorder (OUD) remains a major public health crisis in the United States, with significant challenges in treatment access, retention, and workforce capacity. OUD care teams, including addiction medicine physicians and peer recovery coaches (PRCs), support patients receiving medication for OUD (MOUD), yet face heavy workloads and burnout. Artificial intelligence (AI) innovations, particularly large language model (LLM)–powered chatbots, may extend PRC support and provide patients with on-demand recovery support between clinic visits and PRC contacts. However, evidence on their development, feasibility, acceptability, and usability in addiction services is limited.

Objective:

To describe the development, feasibility, acceptability, and usability of an AI-powered health coaching chatbot (“Suzy”) designed to support patients in OUD recovery.

Methods:

Clinicians, researchers, and technology developers conducted a multiphase study. In the formative phase, we conducted focus groups and interviews with 12 health care professionals and 8 patients with substance use histories to specify chatbot functions and then developed a rule-based chatbot. In the pilot phase, we conducted usability testing on the rule-based chatbot with 8 patients reporting substance use, who completed standardized tasks, surveys, and qualitative interviews. Measures included the System Usability Scale (SUS), Net Promoter Score (NPS), and Single Ease of Use Question (SEQ). In the LLM development phase, we developed a LLM chatbot co-designed with PRCs, OUD recovery patients and other substance use experts.

Results:

Chatbot functions included craving management, appointment reminders, and resource referrals. All usability testing tasks were completed, supporting feasibility. Quantitative and qualitative feedback indicated strong acceptability and usability with an average SUS score of 93 (benchmark 68), NPS of 63 (benchmark 35), and mean SEQ score of 6.5/7. Patients valued Suzy’s approachable, nonjudgmental language, and features that promoted accountability and self-reflection and 24/7 availability, while emphasizing that chatbots should supplement but not replace human support. The LLM-enhanced chatbot development emphasized safety, accuracy, safety escalation protocols to mitigate risks of inappropriate chatbot responses, human-in-the-loop features, and expanded conversational flexibility and personal tailoring.

Conclusions:

A rule-based chatbot, designed to support OUD care, demonstrated strong feasibility, usability, and acceptability. LLM chatbot development required more robust safety and emergency reporting features while having more patient-responsive conversational functions. By providing on-demand coaching, referrals, and reminders, Suzy may extend the reach of care teams, alleviate provider burden, and enhance patient engagement. Additional work is needed to understand how to best integrate Suzy into the patient’s recovery journey to ensure human support remains accessible and prioritized. Next steps include evaluating the use of Suzy after LLM integration in real-world settings and its efficacy in reducing substance use.