Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Nov 23, 2022
Date Accepted: Mar 31, 2023
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Evaluating Conversational Agents for Mental Health: A Scoping Review of Outcome Measurement Instruments
ABSTRACT
Background:
Rapid proliferation of mental health interventions delivered through conversational agents (CA) calls for high-quality evidence to support their implementation and adoption. The choice of outcomes, outcome measurement instruments and assessment modalities play an important role in ensuring the relevant and high-quality evaluation of such interventions.
Objective:
We aimed to identify the types of outcomes, outcome measurement instruments, and assessment modalities used to assess the clinical, user experience, and technical outcomes in studies that evaluated the effectiveness of CA interventions for mental health.
Methods:
We undertook a scoping review of the relevant literature to review the types of outcomes, outcome measurement instruments, and the assessment platform in studies that evaluated the effectiveness of CA mental health interventions. We performed a comprehensive search of electronic databases up to April 2021.
Results:
We included 32 studies that targeted the promotion of mental well-being (17/32, 53%) and the treatment and monitoring of mental health symptoms (15/32, 66%). The studies reported 203 outcome measurement instruments used to measure clinical (123/203, 61%), user experience outcomes (75/203, 37%), technical outcomes (2/203, 1%), and other outcomes (3/203, 1%). Most of the outcome measurement instruments were used only in one study (150/203, 74%), were self-reported questionnaires (170/203, 84%) and most were delivered electronically via survey platforms (61/203, 30%). No validity evidence was cited for more than half of the outcome measurement instruments (107/203, 53%), which were largely created or adapted for the study in which they were used (95/107, 89%).
Conclusions:
The diversity of outcomes and the choice of outcome measurement instruments employed in studies on CAs for mental health points to the need for an established minimum core outcome set and greater use of validated instruments. Future studies should also capitalize on the affordances made available by the CAs and smartphones to streamline the evaluation and reduce participants’ input burden inherent to self-reporting.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.