Accepted for/Published in: JMIR Formative Research
Date Submitted: Jun 15, 2024
Open Peer Review Period: Jul 3, 2024 - Aug 28, 2024
Date Accepted: Oct 10, 2024
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Towards Meaningful Evaluations of Algorithmic Monitoring in eMental Health: The Case of an Online Grief Service for Older Mourners
ABSTRACT
Background:
Artificial intelligence (AI) tools hold much promise for mental healthcare by increasing the scalability and accessibility of care. However, current development and evaluation practices of AI tools in mental healthcare limit the meaningfulness of their evaluation for healthcare contexts and thereby, the practical usefulness of such tools for professionals and clients alike.
Objective:
To move towards meaningful evaluation of AI tools in eMental health, this article demonstrates the evaluation of an AI monitoring tool that detects the need for more intensive care in an online grief intervention for older mourners.
Methods:
We take a three-fold evaluation approach (1) using the F1-metric to evaluate the tool’s capacity to classify user monitoring parameters, including affect, as (a) in need of more intensive support, or (b) recommendable to continue using the online grief intervention as is; (2) using linear regression to assess the predictive value of users’ monitoring parameters for clinical changes in grief, depression, and loneliness over the course of a 10-week intervention. Finally, (3) we collect qualitative experience data from eCoaches (N=4) who incorporated the monitoring in their weekly e-mail guidance during the 10-week intervention.
Results:
(1) Based on N=174 binary recommendation decisions, the F1-score of the monitoring tool was 0.91. (2) Due to minimal variation in depression and loneliness scores after the 10-week intervention compared to before the intervention, only one linear regression was conducted with the difference score in grief before and after the intervention as dependent variable and participants’ mean score on the monitoring assessment tool, the estimate and slope of growth curves fitted to each participant’s response pattern to the monitoring assessment tool as predictors. Only the mean score exhibited predictive value for the observed change in grief (R2 =1.19, SE 0.33, t(df) = 3.58(16), P=.002). (3) The eCoaches appreciated the monitoring tool as a) an opportunity to confirm their impression about the participant based on a clinical interview prior to the intervention, b) a source for personalizing their e-mail guidance and c) an opportunity to detect when participants’ mental health deteriorated during the intervention.
Conclusions:
Each evaluation approach used in this article comes with its own set of limitations and challenges, including (a) skewed class distributions in prediction tasks based on real-life mental health data and (b) choosing meaningful statistical analyses based on clinical trial designs not targeted at evaluating AI tools. However, using multiple evaluation methods provides a good basis for drawing clinically meaningful conclusions and recommendations for improving the clinical value of any specific AI monitoring tool for its intended clinical context.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.