Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.
Who will be affected?
Readers: No access to all 28 journals. We recommend accessing our articles via PubMed Central
Authors: No access to the submission form or your user account.
Reviewers: No access to your user account. Please download manuscripts you are reviewing for offline reading before Wednesday, July 01, 2020 at 7:00 PM.
Editors: No access to your user account to assign reviewers or make decisions.
Copyeditors: No access to user account. Please download manuscripts you are copyediting before Wednesday, July 01, 2020 at 7:00 PM.
Agreement between experts and an untrained crowd for identifying dermoscopic features: a feasibility study
Jonathan Kentley;
Jochen Weber;
Konstantinos Liopyris;
Ralph P Braun;
Ashfaq A Marghoob;
Elizabeth A Quigley;
Kelly Nelson;
Kira Prentice;
Erik Duhaime;
Allan C Halpern;
Veronica Rotemberg
ABSTRACT
Background:
Dermoscopy is commonly used for the evaluation of pigmented lesions, however agreement between experts for identification of dermoscopic structures is known to be relatively poor. Expert labelling of medical data is a bottleneck in the development of diagnostic machine learning (ML) tools, and crowdsourcing has been demonstrated as a cost- and time-efficient method for the annotation of medical images.
Objective:
-
Methods:
-
Results:
We obtained 139,731 ratings of six dermoscopic “super-features” from an untrained crowd with comparable repeatability to a group of dermatologists and good or excellent agreement against a thresholded average expert.
Conclusions:
This confirms the feasibility and dependability of using a non-expert crowd as a scalable solution to annotate large sets of dermoscopic images with several potential clinical and educational applications, including the development of novel, interpretable, ML tools.
Citation
Please cite as:
Kentley J, Weber J, Liopyris K, Braun RP, Marghoob AA, Quigley EA, Nelson K, Prentice K, Duhaime E, Halpern AC, Rotemberg V
Agreement Between Experts and an Untrained Crowd for Identifying Dermoscopic Features Using a Gamified App: Reader Feasibility Study