JMIR Publications

ABSTRACT

Background:

Screening large bibliographic datasets is resource intensive, and advent of large language models have paved the way for automated screening for literature reviews. However, the LLM based screening requires transparent validation against human decisions.

Objective:

To validate the efficacy and reliability of AI-based screening automation in performing the first-pass (FPS) and full-text review (FTR) against human screening decisions.

Methods:

This retrospective internal validation of LLM-based screening using AccuScreener, a module of SymproAI, included five completed evidence-review datasets. At first step, citation deduplication was performed and compared between AccuScreener and human analysts. Raw AI decisions were compared directly with the corresponding final manual FPS and FTR reference decisions. At FPS, the analysis focused on accuracy, specificity, and negative predictive value (NPV); sensitivity and positive predictive value (PPV) were major assessment parameters at FTR. Replicability of screening decisions and structured reason categories was evaluated at both stages.

Results:

AccuScreener identified a larger number of putative duplicate citations than human analysts across all reviews: 2.53% vs 2.34% in SLR-1, 3.64% vs 3.23% in SLR-2, 4.74% vs 4.34% in SLR-3, 9.11% vs 8.93% in SLR-4, and 5.91% vs 5.63% in the composite TLR, and completed deduplication within 10–30 seconds per project. At FPS, AccuScreener demonstrated 89.3%–98.8% accuracy, 89.4%–99.2% specificity, and 96.1%–99.7% NPV across five completed reviews. In an exploratory analysis of 664 paired FPS records with missing abstracts, sensitivity was 46.0%, specificity was 91.0%, and accuracy was 86.7%. At FTR, sensitivity ranged from 78.9% to 100% and PPV from 58.6% to 90.1%, with two reviews reaching 100% sensitivity. Across three reviews, complete three-run agreement was 98.4% for decisions and 76.3% for structured reason categories at FPS; corresponding FTR agreement was 98.3% and 93.3%, respectively.

Conclusions:

AccuScreener showed high overall agreement with human-reference decisions across the complete FPS and FTR datasets. The findings support automation-assisted screening with manual-review safeguards particularly when the citation information is insufficient. Prospective validation across additional review types is warranted.