JMIR Publications

ABSTRACT

Background:

Consumer-facing digital health interventions provide a promising avenue to bridging gaps in mental health care delivery. To evaluate these interventions, understanding how the target population uses a solution is critical to the overall validity and reliability of the evaluation. As a result, usage data (analytics) can provide a proxy to evaluating engagement of a solution. However, there is a paucity of guidance on how usage data or analytics should be used to assess and evaluate digital mental health interventions.

Objective:

To examine how usage data are collected and analyzed in evaluations on mental health mobile applications for transition-aged youth.

Methods:

A scoping review was conducted using the Arksey and O’Malley framework. A systematic search was conducted from five journal databases using keywords related to usage/engagement, mental health apps and evaluation. A total of 1,784 articles from 2008–2019 were identified and screened to ensure they included analytics and evaluated a mental health app for transition-aged youth. After full-text screening, 49 articles were included for analysis.

Results:

Of the 49 articles included for analysis, 40 unique digital mental health innovations were evaluated and more than 80% of the articles were published over the past six years. About 80% involved a randomized controlled trial (RCT) and evaluated apps with information delivery features. There were heterogeneous findings in the concept that analytics was ascribed to, with the top three being engagement, adherence and acceptability. There was also significant spread in the number of metrics collected by each study with 35% (17/49) of the articles reported collecting only one metric and 29% (14/49) collecting four or more analytic metrics. The number of modules completed, session duration, and number of logins were the most common usage metrics collected.

Conclusions:

This review of the current literature identified significant variability and heterogeneity in using analytics to evaluate digital mental health interventions for transition-aged youth. The large proportion of publications from the last six years suggests that user analytics are increasingly being integrated into evaluation of these apps. Numerous gaps related to selecting appropriate and relevant metrics, and defining successful or high levels of engagement have been identified for future exploration. While long-term use or adoption is an important precursor to realizing the expected benefits of an app, few studies have examined this issue. Researchers would benefit from clarification and guidance on how to measure and analyze app usage in terms of evaluating digital mental health interventions for transition-aged youth. Given the established role of adoption and the success of health information technologies, understanding how to abstract and analyze user adoption for consumer digital mental health apps is also an emerging priority.