Accepted for/Published in: JMIR Medical Education
Date Submitted: Sep 18, 2020
Date Accepted: Apr 4, 2021
Impact of systematic review automation tools on methodological quality and time taken to complete systematic review tasks: A case study
ABSTRACT
Background:
Systematic reviews are considered the highest level of evidence to answer research questions; however, they are time and resource intensive.
Objective:
For systematic review tasks in which an systematic review automation tool has been developed, this study seeks to determine the difference in time to complete the task and the error rate of the task when completed manually using current standard methods versus automated using systematic review automation tools.
Methods:
A case study examining the methods used to undertake a systematic review of randomised controlled trials which delivered a healthcare intervention to adults with chronic kidney disease was conducted. Two researchers (manual team) conducted the systematic review using current methods, a total of 16 tasks. Another two researchers (automation team) conducted the tasks where a systematic review automation tool was available, a total of six tasks. The time taken and error rate of the six tasks which were completed by both teams were compared.
Results:
The approximate time taken for the manual team to produce a draft of the background, methods, and results section of the systematic was 126 hours. For the six tasks in which times were compared, the manual team used 2493 minutes (42 hours), compared to 708 minutes (12 hours) by the automation team. The manual team had a higher error rate in the “Run systematic search” and “Assess Risk of Bias” tasks. The automation team had a higher error rate for the “Deduplicate search results” task. Error rates were similar for the “Screening titles and abstracts” and “Screening the full texts” tasks. Barriers were using EndNote libraries in a cloud environment; geographical dispersal of team members; and accessibility to systematic review automation tools. The facilitators were review topic knowledge; access to review methodology support staff; detailed systematic review automation tool user guides; and systematic review automation experience of colleagues.
Conclusions:
For tasks required to complete a systematic review, the time taken to complete the tasks in which a systematic review automation tool existed was reduced whilst methodological quality was maintained. This case study may increase confidence in the use of systematic review automation tools among systematic reviewers.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.