JMIR Publications

ABSTRACT

Background:

The use of ecological momentary assessment (EMA) designs have been on the rise in mental health epidemiology. However, there is a lack of knowledge on the determinants of participation and compliance in EMA studies, reliability of measures, and underreporting of methodological details and data quality indicators.

Objective:

To evaluate the quality of EMA data in a large sample of university students by estimating participation rate and mean compliance, identifying predictors of individual-level participation and compliance, evaluating between- and within-person reliability of measures of negative and positive affect, and identifying potential careless responding.

Methods:

A total of 1,259 university students were invited to participate in a 15-day EMA study on mental health problems. Logistic and Poisson regression investigated the associations of sociodemographic factors, lifetime adverse experiences, 12-month stressful events, and mental disorder screens with EMA participation and compliance. Multilevel reliability and intraclass correlation coefficients were obtained for positive and negative affect measures. Careless responders were identified based on low compliance or individual reliability coefficients.

Results:

Of those invited, 782 (62.1%) participated in the EMA study with a mean compliance of 76.9%. Participation was higher among females (OR= 1.41, CI95= 1.06-1.87) and lower among those aged 30 or more (OR= 0.20, CI95= 0.08-0.43; vs age 18-21), those who had experienced the death of a friend of family member in the past 12-month (OR= 0.73, CI95= 0.57-0.94), and a 12-month suicide attempt (OR=0.26, CI95= 0.10-0.64). Compliance was particularly low among those exposed to sexual abuse before the age of 18 (exp[b] = 0.87) or to sexual assault or rape in the past year (exp[b] = 0.80) and among those with 12-month positive alcohol use disorder screens (exp[b]= 0.89). Between-person reliability of negative and positive affect was strong (RkRn >0.97), while within-person reliability was fair to moderate (Rcn> 0.43). Of all answered assessments, 0.86% were flagged as careless responses because time of response per item was below 1 second, or they gave the same response to all items. Of participants, 17.5% could be considered careless responders due to low compliance (<44.6%), or very low to null individual reliability (Raw α< 0.11) in either negative or positive affect.

Conclusions:

Data quality assessments should be carried out in EMA studies in a standardized manner to provide robust conclusions to advance the field. Future EMA research should implement strategies to mitigate non-response bias as well as conduct sensitivity analyses to assess possible exclusion of careless responders. Clinical Trial: Registration: osf.io/p7csq