JMIR Publications

ABSTRACT

Background:

Obtaining a diagnosis of neuropsychiatric disorders such as autism requires long waiting times often exceeding a year. Furthermore, obtaining a professional diagnosis is often prohibitively expensive for much of the global population. Crowdsourcing provides a cheap and possibly free alternative that can allow underserved populations to obtain an accurate diagnosis.

Objective:

To determine whether paid crowd workers on Amazon Mechanical Turk (AMT) as well as citizen crowd workers on a public website shared on social media can provide accurate online screening for autism from watching a short video clip.

Methods:

Three online studies: (1) a paid crowdsourcing task on AMT (N=55) where crowd workers were asked to rate ten short video clips of children as having “autism” or “no autism”, (2) a follow-up, more complex paid crowdsourcing task on AMT (N=27) with only those raters who correctly rated 8 or more of the 10 videos during the first study, and (3) a public unpaid “citizen healthcare” study (N=71) with the same task as in the first study. 50% of the videos in all studies contained children with autism, 50% of videos contained females, and the age range of children in the videos ranged from 2 to 5 years.

Results:

Study 1: The mean score of the participants who completed all questions was 7.50 (SD = 1.46) out of 10. When only analyzing the workers who scored reasonably well (scored 8 out of 10 or higher; n=27 out of 55), there was a weak negative correlation between the time spent rating the videos and the sensitivity (r = -0.44, p = 0.02). Study 2: The mean score of the participants rating new videos was 6.76 (SD = 0.59) out 10. Across all videos, the average deviation between the average crowdsourced answers and gold standard ratings provided by two expert clinical research coordinators was 0.557, with a standard deviation of 0.51 (the maximum possible deviation is 3). All paid crowdworkers who scored an 8 out of 10 in Study 1 either enjoyed performing the task in Study 2 or provided no negative comments. Study 3: There was a negative correlation between whether the rater is a parent and sensitivity (r = -0.26, p=0.02) as well as precision (r = -0.28, p=0.02). The mean score of the participants who completed all questions was 6.61 (SD = 1.60) out of 10. A two-tailed t-test between the scores of the paid workers in Study 1 and the unpaid workers in Study 3 showed significance in the difference (p=0.0009).

Conclusions:

(1) Many paid crowdworkers on AMT enjoy answering screening questions from videos, suggesting higher intrinsic motivation to make quality assessments.”). (2) Both paid crowdsourcing and citizen crowdsourcing provide promising screening assessments of pediatric autism comparable to that of professional raters. (3) Parents of children with autism likely overfit their intuition to their own affected child, which is problematic for a disorder such as autism which has a wide range of phenotypic manifestations. (4) Paid workers will outperform a general unpaid crowd of online citizen workers.