JMIR Publications

ABSTRACT

Background:

Researchers have many options for online survey data collection, ranging from access to curated probability-based panels, where individuals are selectively invited to join based on their membership in a representative population, to convenience panels, which are open for anyone to join. The mix of respondents available also varies greatly regarding representation of a population of interest and in motivation to provide thoughtful and accurate responses. Despite the additional dataset-building labor required of the researcher, convenience panels are much less expensive than probability-based panels. However, it is important to understand what may be given up in terms of data quality for those cost savings.

Objective:

This study examined the relative costs and data quality from fielding equivalent surveys on Amazon’s Mechanical Turk (MTurk), a convenience panel that anyone 18 years or older with internet access and an electronic device can join, and on KnowledgePanel, a nationally representative probability-based panel whose members have been selectively invited to join.

Methods:

We administered the same survey measures to MTurk (2021) and KnowledgePanel (2022) members. We applied several quality assurance steps recommended in the literature at the time to enhance the data quality achieved using MTurk. Ipsos, the owner of KnowledgePanel, followed their usual (industry standard) protocols. The survey was designed to support psychometric analyses and included over 60 items from the Patient-Reported Outcomes Measurement Information System© (PROMIS®), demographics, and a list of health conditions. Those who indicated current back pain also received several established instruments used to measure back pain impact. Two fake conditions (“Syndomitis” and “Chekalism”) were used to identify those more likely to be honest respondents. We examined the quality of each platform’s data using several recommended metrics (e.g., consistency, reliability, representativeness, missing data, correlations) including and excluding those respondents who endorsed a fake condition.

Results:

We found that prescreening in the MTurk sample (removing those who endorsed a fake health condition) improved data quality, but KnowledgePanel data quality generally remained superior. While MTurk’s unweighted point estimates for demographics displayed the usual mismatch with national averages (younger, better educated, lower income), weighted MTurk data matched national estimates. KnowledgePanel’s point estimates better matched national benchmarks even before nonresponse weighting. Correlations between PROMIS measures and age and income were similar between MTurk and KnowledgePanel. However, correlations between PROMIS measures and education were dramatically different; the direction of over half of these relationships in the MTurk sample was opposite than expected from theory. Therefore, caution is needed if using MTurk for studies where education is a key variable.

Conclusions:

The data quality of our MTurk sample was often inferior to that of KnowledgePanel, but not so much as to negate the benefits of its cost savings for some uses. Clinical Trial: ClinicalTrials.gov NCT04426812