JMIR Publications

ABSTRACT

Background:

Evaluating digital interventions using remote methods enables the recruitment of large numbers of participants relatively conveniently and cheaply compared with in-person methods. However, conducting research remotely, based on participant self-report with little verification, is open to automated “bots” and participant deception.

Objective:

This paper uses a case study of a remotely conducted trial of an alcohol reduction app to highlight and discuss i) the issues with participant deception affecting remote research trials with financial compensation, and ii) the importance of rigorous data management to detect and address these issues.

Methods:

A randomised controlled trial (n=5,602) evaluating the effectiveness of an alcohol reduction app, Drink Less, recruited participants online from July 2020 to March 2022. Follow-up occurred at three time points with financial compensation offered (up to £36). Address authentication and telephone verification were used to detect two kinds of deception: “bots”, i.e. automated responses generated in clusters, and manual participant deception, i.e. participants providing false information.

Results:

Of the 1,142 participants who enrolled in the first two months of recruitment, 75.6% (n=863) were identified as bots during data screening. As a result, a CAPTCHA was added and after this, no more bots were identified. Manual participant deception occurred throughout the study. Of 5,956 (excluding bots) who enrolled in the study, 298 (5.0%) were identified as false participants. The extent of this decreased from 110 in November 2020 to a negligible level by February 2022 including a number of months with 0. The decline occurred after we added further screening questions such as attention checks, removed the prominence of financial compensation from social media advertising, and an additional requirement to provide a mobile phone number for identity verification.

Conclusions:

Data management protocols are necessary to detect automated bots and manual participant deception in remotely conducted trials. Bots and manual deception can be minimised by adding a CAPTCHA, attention checks and a requirement to provide a phone number for identity verification, and not prominently advertising financial compensation on social media. Clinical Trial: ISRCTN Number: ISRCTN64052601 https://www.isrctn.com/ISRCTN64052601