Accepted for/Published in: JMIR Formative Research
Date Submitted: May 11, 2022
Date Accepted: Oct 20, 2022
Data Quality and Study Compliance among College Students across Two Recruitment Sources: A Two-study Investigation
ABSTRACT
Background:
Participants who do not fully process survey instructions can reduce a study’s power and hinder generalizability. Common concerns among researchers using self-report measures are data quality and participant compliance. Attrition can similarly hurt power and generalizability.
Objective:
Given that college students comprise the majority of samples in psychological studies, especially examinations of student issues and psychological health, it is critical to understand how college student recruitment sources impact data quality (operationalized as attention check items with directive instructions and correct answers) and retention (operationalized as completion of follow-up surveys over time).
Methods:
The current examination was a follow-up analysis on two previously-published studies to explore data quality and study compliance Study 1 was a cross-sectional, online survey examining college stressors and psychological health (N = 407; 76% female; 57% White, 28% Black; Mage = 22.65). Study 2 was a longitudinal college drinking intervention trial with an in-person baseline session and two online follow-up surveys (N = 528; 72% female; 41% White, 53% Black; Mage = 19.85). Attention checks were included in both studies to assess data quality. Both studies recruited from a) a psychology participation pool, and b) the general student body.
Results:
A greater proportion of participants recruited through the psychology pool failed attention checks for both studies, suggesting poorer data quality. The psychology pool was also associated with lower retention rates over time. After screening out those who failed attention checks, some correlations among study variables were stronger, some were weaker, and some were fairly similar, potentially suggesting bias introduced by including these participants. Differences among indicators of internal consistency for study measures were negligible. Finally, Attention check failure was not significantly associated with most demographic characteristics, but varied across some racial identities. This suggests filtering out cases who failed attention checks may not limit sample diversity.
Conclusions:
Investigators conducting college student research should carefully consider recruitment, and should include attention checks or other means of detecting poor quality data. Recommendations for researchers are discussed. Clinical Trial: ClinicalTrials.gov NCT03440463
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.