Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Jan 9, 2020
Date Accepted: Sep 20, 2020
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Predicting Dropout in Digital Health Interventions: A Process to Analyze User Journey Data
ABSTRACT
Background:
User dropout is a widespread concern in the delivery and evaluation of digital health (i.e., web- and mobile application) interventions. Researchers have yet to fully realize the potential of the large amount of data generated by these technology-based programs. Of particular interest is the ability to predict who will drop out of an intervention. This may be possible through the analysis of user journey data – self-reported as well as system generated data produced by the path (or journey) an individual takes to navigate through a digital health intervention.
Objective:
The purpose of this study is to provide a step-by-step process for the analysis of user journey data and eventually to predict dropout in the context of digital health interventions. The process is applied to data of an Internet-based intervention for insomnia as a way to illustrate its use.
Methods:
Steps of user journey analysis, including data transformation, feature engineering, and statistical model analysis and evaluation, are presented. Dropouts were predicted based on data of 151 participants from a fully automated web-based program (SHUTi) that delivers cognitive behavioral therapy for insomnia. Various machine learning techniques were used and evaluated based on their predictive performance. Relevant features from the data are reported that predict user dropout.
Results:
Accuracy of predicting dropout (AUC values) varied depending on point in time of prediction and the machine learning technique. After model evaluation, boosted decision trees achieved AUC values ranging between 0.6-0.9. Additional handcrafted features such as the time to complete certain steps of the intervention, time to get out of bed, and days since last interaction with the system contributed to the prediction performance.
Conclusions:
The results support the feasibility and potential of analyzing user journey data in order to predict dropout. Theory driven handcrafted features increased prediction performance. The ability to predict dropout on an individual level could be used to enhance decision-making for researchers and clinicians as well as inform dynamic intervention regimens.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.