JMIR Publications

ABSTRACT

Background:

Using mobile health technology has sparked a broad engagement of data science and machine learning methods to leverage the complex, assorted amount of data for mental health purposes. Despite many studies, there is a reported underdevelopment of user engagement concepts, and the desire for high accuracy or significance has shown to lead to low explicability and irreproducibility.

Objective:

To overcome such reasons of poor analysis input and facilitate the reproducibility and credibility of artificial intelligence applications, we aim to explore principal characteristics of user interaction with digital mental health.

Methods:

We generated five latent features based on previous research, expert opinions from digital mental health, and informed by data. The features were analyzed with descriptive statistics and data visualization. We carried out two rounds of evaluations with data from 12,400 users of IntelliCare, a mental health platform with 12 apps. First, we focused to proof concept and second, we assessed reproducibility by drawing conclusion from distribution differences. User data was drawn from both research trials and public deployment on Google Play.

Results:

Our algorithms showed advantages over commonly used concepts and reproduce in our public data set with different underlying behavioral strategies. These measures relate to the distribution of a user’s allocated attention, users’ circadian behavior, their consecutive commitment to a specific strategy, and users’ interaction trajectory. Because distributions between research trial and public deployment were similar, consistency was implied regarding the underlying behavioral strategies: psychoeducation and goal setting are used as a catalyst to overcome the users’ primary obstacles, sleep hygiene is addressed most regularly, while regular self-reflective thinking is avoided. Relaxation as well as cognitive reframing have increased variance in commitment among public users, indicating the challenging nature of these apps. The relative course of users’ engagement is similar in research and public data.

Conclusions:

We argue that deliberate, a-priori feature engineering is essential for reproducible, tangible, and explainable study analyses. Our features enable improved results as well as interpretability, providing an increased understanding of how people engage with multiple mental health apps over time. Since we based the generation of features on generic interaction, these methods are applicable to further methods of study analysis and digital health.