Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Apr 17, 2025
Date Accepted: Dec 15, 2025
Which Phases of Living Evidence Synthesis Use Artificial Intelligence (AI): Living Evidence Synthesis (Version 1)
ABSTRACT
Background:
Living evidence (LE) synthesis refers to the method of continuously updating systematic evidence reviews to incorporate new evidence. It has emerged to address the limitations of the traditional systematic review process, particularly the absence or delays in update publication. The emergence of COVID-19 accelerated the progress in the field of LE synthesis, and currently, the applications of artificial intelligence (AI) in LE synthesis are expanding rapidly. However, in which phases of LE synthesis should AI be used remains an unanswered question.
Objective:
(ⅰ) To document which phases of LE synthesis where AI is used. (ⅱ) To investigate whether AI improves the efficiency, accuracy, or utility of LE synthesis.
Methods:
We searched Web of Science, PubMed, the Cochrane Library, Epistemonikos, the Campbell Library, IEEE Xplore, medRxiv, COVID-19 Evidence Network to support Decision-making (COVID-END), and McMaster Health Forum. We used Covidence to facilitate the monthly screening and extraction processes to maintain the LE Synthesis process. Studies that used or developed AI/semi-automated tools in the phases of LE synthesis were included.
Results:
A total of 24 studies were included, including 17 on LE syntheses, with four involving tool development, and seven on living meta-analyses, with three involving tool development. (ⅰ) 20 studies applied AI/semi-automated tools in the data extraction/collection and risk of bias assessment phase, and only one addressed the update publication phase. (ⅱ) A total of 34 AI/semi-automated tools were involved, comprising 12 AI tools and 22 semi-automated tools. The most frequently used AI/semi-automated tools were machine learning classifiers (n=5) and the Living Interactive Evidence synthesis platform (n=3). (ⅲ) Three studies demonstrated the improvement in efficiency achieved based on time, workload, and conflict rate metrics. Nine studies applied AI/semi-automated tools in LE synthesis, obtaining a mean recall rate of 96.24%, and six studies achieved a mean F1 score of 92.17%. Additionally, eight studies reported precision values ranging from 0.2% to 100%.
Conclusions:
Artificial intelligence and semi-automated tools primarily facilitate data extraction/collection and risk of bias assessment. The use of AI/semi-automated tools in LE synthesis improves efficiency, leading to high accuracy, recall, and F1 scores, while precision varies across tools. Clinical Trial: https://doi.org/10.17605/OSF.IO/4FVDQ
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.