JMIR Publications

ABSTRACT

Background:

The concept of engagement surfaces as a predictor for effectiveness of eHealth interventions. However, a shared understanding of what engagement is and how to measure it, is missing. This makes research in this area fragmented and hard to generalize, and implies that there is a need for a valid way to measure engagement. Therefore, a new scale has been developed; the TWente Engagement with Ehealth Technologies Scale (TWEETS). This scale was developed based on interviews with engaged health app users and employs a definition of engagement that incorporates behavior, cognition and affect, as is common in other fields of research where engagement is used.

Objective:

The current paper is aimed at evaluating the psychometric properties of the newly developed TWEETS. As a secondary objective, the psychometric properties of the TWEETS were compared to those of the experiential part of the Digital Behavior Change Intervention engagement scale (DCBI-E).

Methods:

Participants (n = 288) were asked to use any step-counter app on their smartphone for two weeks. They were asked to fill out an online questionnaire at four time points (T0 = baseline, T1 = after 1 day, T2 = 1 week and T3 = 2 weeks). Scales on engagement, personality, involvement, enjoyment, usage and perceived behavior change were included. Internal consistency and reliability; and convergent, divergent, and predictive validity were assessed of both engagement scales.

Results:

On internal consistency, Cronbach's alpha of the TWEETS was 0.86, 0.86 and 0.87 on T1, T2 and T3 respectively. Exploratory factor analyses of the TWEETS indicated that a one-factor structure best fitted the data. Analyses on convergent validity showed that the TWEETS is moderately to strongly correlated with involvement and enjoyment, which can be seen as related to cognitive and affective engagement, respectively. Results on convergent The TWEETS showed significant moderate to strong correlations with involvement and enjoyment, and no to weak correlations with use frequency. Correlations between the engagement measure and frequency of use were non-significant or small, while differences between adherers and non-adherers on engagement were significant (p<.001), but small . This indicates that engagement is related to usage, but much less so than is often assumed. Pearson's correlations between conscientiousness and intellect/imagination, and the TWEETS at the different time points were non-significant, indicating divergent validity. Lastly, the TWEETS demonstrated predictive validity. Engagement measured at T1 was predictive of perceived behavior change at T3, with an explained variance of 16%. Comparing the psychometric properties of the TWEETS and the DCBI-E showed that these properties were comparable on some aspects (e.g. internal consistency) and on other aspects the TWEETS was somewhat superior (divergent and predictive validity).

Conclusions:

The TWEETS performs quite well as a one-factor engagement measure: the scale showed high internal consistency, reasonable test-retest reliability and convergent validity, good divergent validity, and reasonable predictive validity. Further research is needed to replicate these findings in other target groups and technologies, but the TWEETS seems to be valuable addition to the toolbox of eHealth researchers.