Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Accepted for/Published in: Journal of Medical Internet Research

Date Submitted: Jul 14, 2020
Date Accepted: Nov 30, 2020
Date Submitted to PubMed: Dec 2, 2020

The final, peer-reviewed published version of this preprint can be found here:

Deep Neural Network for Reducing the Screening Workload in Systematic Reviews for Clinical Guidelines: Algorithm Validation Study

Yamada T, Yoneoka D, Hiraike Y, Hino K, Toyoshiba H, Shishido A, Noma H, Shojima N, Yamauchi T

Deep Neural Network for Reducing the Screening Workload in Systematic Reviews for Clinical Guidelines: Algorithm Validation Study

J Med Internet Res 2020;22(12):e22422

DOI: 10.2196/22422

PMID: 33262102

PMCID: 7806440

Deep neural network-based machine learning reduces the screening workload for systematic review: Investigation based on recent clinical guidelines

  • Tomohide Yamada; 
  • Daisuke Yoneoka; 
  • Yuta Hiraike; 
  • Kimihiro Hino; 
  • Hiroyoshi Toyoshiba; 
  • Akira Shishido; 
  • Hisashi Noma; 
  • Nobuhiro Shojima; 
  • Toshimasa Yamauchi

ABSTRACT

Background:

Performing systematic reviews is a time-consuming and resource-intensive process.

Objective:

We investigated whether a machine learning system could perform systematic reviews more efficiently.

Methods:

All systematic reviews and meta-analyses of interventional randomized controlled trials cited in recent clinical guidelines from the American Diabetes Association, American College of Cardiology, American Heart Association (x2), and American Stroke Association were assessed. After reproducing the primary screening dataset according to the published search strategy, we extracted correct articles (actually reviewed) and incorrect articles (not reviewed) from the dataset. Next, these two sets of articles were used to train a neural network-based artificial intelligence engine (Concept Encoder, FRONTEO Inc.). The primary endpoint was work saved over sampling at 95% recall.

Results:

Among 145 candidates, 8 reviews of randomized controlled trials fulfilled the inclusion criteria. Using these 8 reviews, the machine learning system significantly reduced the literature screening workload by at least 6-fold versus manual screening. When machine learning was initiated using two correct articles randomly selected by a researcher, it achieved a 10-fold reduction in workload versus manual screening with high sensitivity for eligible studies. The area under the receiver operating characteristics curve increased dramatically every time the algorithm learned a correct article.

Conclusions:

The Concept Encoder achieved a 10-fold or better reduction of the screening workload for systematic review after learning two randomly selected studies on the target topic. However, few meta-analyses of randomized controlled trials were included. The Concept Encoder could facilitate the acquisition of evidence for clinical guidelines. Clinical Trial: UMIN000032663


 Citation

Please cite as:

Yamada T, Yoneoka D, Hiraike Y, Hino K, Toyoshiba H, Shishido A, Noma H, Shojima N, Yamauchi T

Deep Neural Network for Reducing the Screening Workload in Systematic Reviews for Clinical Guidelines: Algorithm Validation Study

J Med Internet Res 2020;22(12):e22422

DOI: 10.2196/22422

PMID: 33262102

PMCID: 7806440

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.