JMIR Publications

ABSTRACT

Background:

Conversational artificial intelligence (AI) is increasingly used to deliver psychotherapy-informed support. Evaluations report dialogue quality, engagement, and symptom change, but these indicators answer different questions, and it is rarely clear which therapeutic work a system performs or whether that work produces change in the people who use it.

Objective:

This scoping review aimed to characterize conversational AI systems for psychotherapy, chart the evaluation evidence they report and how often different kinds of evidence are combined within a study, chart the reporting of ethics-related safeguards, and derive a framework that links delegated therapeutic work to the evidence needed to support it.

Methods:

Following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guideline, we searched PubMed, Embase, Scopus, Web of Science Core Collection, IEEE Xplore, and the ACM Digital Library in December 2025 and ran targeted searches of 3 computing venues. Eligible reports described the design, development, or evaluation of an AI-based conversational system that delivered, supported, or modeled psychotherapy or a psychotherapy-informed intervention. Two reviewers independently screened records and charted data. Reports were classified by primary function (therapeutic delivery, model development, or psychosocial support). Evaluation indicators were charted into 3 dimensions (technical performance, therapeutic process quality, and symptom-related outcomes) and assigned to 7 evidence objects, and the reporting of 4 ethics-related domains was rated.

Results:

Of 18,248 records (10,974 after deduplication), 541 full texts were assessed and 35 reports were included: 18 therapeutic-delivery systems, 9 model-development studies, and 8 psychosocial-support systems. Most were published in 2024 or 2025 (30/35, 86%) and appeared in conference proceedings (24/35, 69%); 26 (74%) used large language models, and 26 (74%) invoked cognitive behavioral therapy. Technical performance was reported in 20 reports, therapeutic process quality in 27, and symptom-related outcomes in 8, all of which described therapeutic-delivery systems; no report combined all 3 dimensions. Therapy-specific competence or fidelity instruments were applied only to simulated or role-played dialogues and model outputs. One study measured a task-proximal user response, 1 measured skills use in daily life, and no charted indicator concerned adverse events or escalation outcomes. Human oversight and accountability were fully reported in 5 of 35 reports and risk management in 7 of the 32 reports to which it applied.

Conclusions:

Evidence on conversational AI for psychotherapy is fragmented across evaluation objects and thins at the steps that link a system’s actions to a person’s change. We propose a framework that pairs the therapeutic work delegated to a system with the evidence needed to evaluate it, separates package efficacy from component attribution and cross-system transfer, and organizes evaluation around the decision delegated to AI, with an evidentiary burden scaled to the consequence of that decision.