JMIR Publications

ABSTRACT

Background:

No established tools are currently capable of quantitatively measuring engagement for health promotion tools, including Digital Therapeutics.

Objective:

We evaluate the engagement index (EI) in commercial health management app for long term use by comparing it with the new EI created by the researchers based on the original EI.

Methods:

Participants were recruited from cancer survivors enrolled in a randomized controlled trial evaluating the impact of mHealth apps on recovery. We used 240 of these patients who were randomly assigned the Noom app. The study validated a new EI compared to an existing EI, with data analysis performed for long-term use. The new EI was calculated based on adapted measurements from the Web Matrix Visitor Index, focusing on click depth, recency, and loyalty indices.

Results:

The old EI demonstrated limited predictive ability for EI values between 6 to 9 months, with a mean squared error (MSE) of .10 and r-squared of .05. However, the new EI displayed enhanced predictive performance. All three new EIs, with different combinations of features, exhibited a lower MSE and higher r-squared compared to the old EI. Cox regression analysis revealed the old EI presented significant hazard ratios (HR) for click depth and loyalty indices, while the new EI consistently demonstrated significant HRs for loyalty and recency indices.

Conclusions:

We evaluated the effectiveness of the EI and proposes potential enhancements due to the ongoing need for a standardized index to measure patient compliance with mHealth applications. We emphasize the importance of log data and suggest avenues for future research to address the subjectivity of the EI and incorporate a broader range of indices for comprehensive evaluation.