Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Currently submitted to: Journal of Medical Internet Research

Date Submitted: Jul 11, 2026

Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.

Leveraging Large Language Models for Systematic Reviews - a Replication Study and Frame-work Proposal

  • Isabel Schnorr; 
  • Katharina S. Appel; 
  • Daniel Maier; 
  • Jörg Janne Vehreschild

ABSTRACT

Background:

Systematic reviews are resource-intensive, and Large Language Models (LLMs) have emerged as promising tools for systematic reviews; however, their ability to replicate complete systematic review processes has been insufficiently evaluated.

Objective:

To determine whether LLMs can accurately replicate key stages of a published systematic review, we assessed performance across the review process and developed a structured framework for transparent and reproducible LLM-assisted systematic reviews.

Methods:

By leveraging Application Programming Interfaces (API), two LLMs (Gemini-2.5-Pro and OpenAI o3) classified titles/abstracts and full texts, and subsequently extracted data from eligible articles. To achieve optimal performance, we iteratively refined the prompt and evalu-ated the classification on inclusion/exclusion criteria level.

Results:

Combined, the two LLMs reached an NPV of 0.99 and FNR of 0.07 with Cohen’s κ of 0.94 (95%-CI=[0.91-0.96]) for title and abstract, and an NPV of 0.95 and FNR of 0.04 with κ of 0.94 (95%-CI=[0.89-0.99]) for full text classification. LLM-based data extraction revealed an overall exact match score of 86.75% after semantic discrepancy resolution.

Conclusions:

Considering the successful replication and the strong agreement after ground-truth reconciliation, we propose the AIMS framework and the llmreviewR software as accessible co-screening tools for applying LLMs in systematic reviews.


 Citation

Please cite as:

Schnorr I, Appel KS, Maier D, Vehreschild JJ

Leveraging Large Language Models for Systematic Reviews - a Replication Study and Frame-work Proposal

JMIR Preprints. 11/07/2026:106373

DOI: 10.2196/preprints.106373

URL: https://preprints.jmir.org/preprint/106373

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.