JMIR Publications

ABSTRACT

Background:

Digital mental health interventions (DMHIs) can help close persistent gaps in access to assessment, prevention, and treatment. Recent advances in generative artificial intelligence (AI), particularly large language models (LLMs), further expand this promise by enabling natural language understanding, personalization, and empathic interaction across assessment, support, and therapeutic contexts. However, significant evaluation challenges persist, including a lack of standardized constructs and validated instruments, which limit the comparability, reproducibility, and generalizability of findings. No systematic synthesis currently documents the frameworks, methodologies, and tools used to evaluate LLMs in DMHIs, thereby hampering the development of a comprehensive evidence base to guide future evaluation efforts.

Objective:

This scoping review aims to systematically map and synthesize the available evidence on frameworks, methodologies, and tools used to evaluate LLMs applied to digital mental health interventions. Specifically, it aims to identify the constructs assessed, the instruments employed, and the evaluation procedures and stages addressed.

Methods:

Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines, this scoping review will search five electronic databases: PubMed, Scopus, Web of Science, IEEE Xplore, and ACM Digital Library, from January 2019 to September 15, 2025. Eligibility criteria will encompass both empirical studies and theoretical proposals evaluating LLMs embedded within DMHIs. Studies limited to risk detection or decision-support systems without an intervention component will be excluded. Data extraction will capture information on conceptual frameworks, methodological designs, evaluation procedures and tools, measured constructs, and other relevant contextual information.

Results:

The systematic search was conducted between September 1 and 15, 2025, yielding 4,273 records across the five databases. After duplicate removal, 2,980 unique records remained for screening. A pilot screening exercise involving four independent reviewers achieved high inter-rater reliability (free-marginal Randolph kappa = 0.81), with 76% unanimous agreement, indicating adequate calibration of selection criteria. These figures are interim process indicators rather than final review findings, as title and abstract screening of the remaining records is currently underway.

Conclusions:

This scoping review is expected to provide one of the first systematic syntheses of frameworks, methodologies, and tools used to evaluate LLMs in digital mental health interventions. By identifying prevailing patterns and gaps, the resulting evidence map is intended to serve as a practical reference for researchers, developers, and policymakers working toward a scientifically grounded, safe, ethical, and effective deployment of LLMs in mental health interventions. Clinical Trial: Open Science Framework (OSF); DOI: 10.17605/OSF.IO/PAZYQ