JMIR Publications

ABSTRACT

Background:

Digital mental health interventions (DMHIs) are increasingly popular for their ability to disseminate evidence-based treatments in a scalable, accessible way. Engagement with digital treatments is critical for effectiveness but remains poorly defined and measured. Most studies rely on subjective self-reports or one or two objective metrics, overlooking the multidimensional nature of how users interact with DMHIs.

Objective:

This exploratory secondary analysis applied a roadmap for comprehensive behavioral engagement measurement, developed for a prior smartphone cognitive behavioral therapy (CBT) program for body dysmorphic disorder, to a 12-week randomized controlled trial of smartphone CBT for obsessive compulsive disorder (OCD) (N=60). Aims were to (1) comprehensively characterize behavioral engagement using 21 objective app analytics-based variables, (2) identify distinct engagement profiles via cluster analysis, (3) assess whether profiles were differentially associated with OCD symptom change, and (4) evaluate associations between profiles and pre-treatment credibility, expectancy, post-treatment satisfaction, and perceived app quality.

Methods:

Behavioral engagement was measured via 21 objective variables derived from app analytics, spanning use volume, frequency, feature utilization, session duration, and temporal patterns. Hierarchical agglomerative cluster analysis identified engagement profiles. Associations between cluster membership and OCD severity change (Yale-Brown Obsessive Compulsive Scale; Y-BOCS) were examined using hierarchical generalized linear mixed models. Objective measures were complemented with subjective self-reports; namely, cluster differences in treatment credibility, expectancy, satisfaction, and perceived app quality were examined using ANOVAs and Wilcoxon signed-rank tests.

Results:

Cluster analysis indicated a three-cluster solution representing high (Cluster 1: n=21, 35%), moderate (Cluster 2: n=31, 52%), and low (Cluster 3: n=8, 13%) app users. All three clusters experienced significant reductions in OCD severity, with no significant differences in pre- to post-treatment Y-BOCS change across clusters. Descriptively, Cluster 1 had a higher rate of treatment responders compared to Cluster 3. While Clusters 1 and 2 had similar perceptions of treatment credibility and expectancy at baseline and satisfaction at post-treatment, Cluster 3 perceived the intervention as significantly less credible, had lower expectations, and reported significantly lower satisfaction compared to Cluster 2. All clusters rated app quality as similarly high.

Conclusions:

Comprehensive behavioral engagement measurement is feasible and informative across different DMHIs and clinical populations. The finding that all engagement profiles showed comparable symptom improvement challenges simple dose-response assumptions and suggests that multiple paths of engagement may be therapeutically beneficial. Pre-treatment credibility and expectancy are associated with engagement patterns, identifying a potential modifiable intervention target. Clinical Trial: Clinicaltrials.gov identifier: NCT04136626