Accepted for/Published in: JMIR Formative Research
Date Submitted: Feb 19, 2024
Date Accepted: Jun 4, 2024
Measuring Engagement in Provider-guided Digital Health Interventions: Development of a Conceptual and Analytical Framework Using Nurse WRITE as an Exemplar
ABSTRACT
Background:
Limited guidance exists for analyzing participant engagement in provider-guided digital health interventions (DHIs). System usage is commonly used, with acknowledged limitations in measuring socio-affective and cognitive aspects of engagement. Nurse WRITE, an online nurse-guided DHI for managing symptoms among women with recurrent ovarian cancer, offers an opportunity to develop a framework for assessing multi-dimensional engagement.
Objective:
This study aims to develop a conceptual and analytic framework to measure socio-affective, cognitive, and behavioral engagement with provider-guided DHIs. We then illustrate the framework’s ability to describe and categorize engagement using Nurse WRITE as an example.
Methods:
A sample of 68 participants from Nurse WRITE who posted on the message boards were included. We adapted a prior framework for conceptualizing and operationalizing engagement across three dimensions and finalized a set of six distinct measures. Utilizing patients' posts, we created two socio-affective engagement measures—total count of socio-affective engagement classes (e.g., sharing personal experience) and total word count—and two cognitive engagement measures—total count of cognitive engagement classes (e.g., asking information-seeking questions) and average response rate. Additionally, we devised behavioral engagement measures using website data—the total count of symptom care plans and plan reviews. K-means clustering categorized the participants into distinct groups based on levels of engagement across three dimensions. Descriptive statistics and narratives were used to describe engagement in three dimensions.
Results:
On average, participants displayed socio-affective engagement 34.7 times, writing 14,851 words. They showed cognitive engagement 19.4 times with a 78.3% response rate to nurses' inquiries. Participants also submitted an average of 1.6 symptom care plans and 0.7 plan reviews. Participants were clustered into high (n = 13), moderate (n = 17), and low engagers (n = 38) based on the six measures. High-engagers wrote a median of 36,956 words. They demonstrated socio-affective engagement approximately 81 times and cognitive engagement around 46 times, approximately six times that of the low engagers and twice that of the moderate engagers. High engagers had a median 91.74% response rate to the nurses’ queries, whereas moderate engagers had 86.36%, and low engagers had 68.32%. High engagers completed a median of three symptom care plans and two reviews, while moderate engagers completed two plans and one review. Low-engagers completed a median of one plan and no reviews.
Conclusions:
This study developed and reported an engagement framework to guide behavioral intervention scientists to understand and analyze participants’ engagement with provider-guided DHIs. Significant variations in engagement levels across three dimensions highlight the importance of measuring engagement with provider-guided DHIs in socio-affective, cognitive, and behavioral dimensions. Future studies should validate the framework with other DHIs, explore the influence of patient and provider factors on engagement, and investigate how engagement influences intervention efficacy.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.