JMIR Publications

ABSTRACT

Background:

Systematic reviews of antimicrobial resistance (AMR) across One Health domains require screening large numbers of records and extracting heterogeneous study characteristics. Evaluations of large language models (LLMs) for these tasks have predominantly focused on proprietary, cloud-hosted systems, raising concerns about cost, data governance and reproducibility. Locally hosted open-weight models offer greater control over these factors, but their performance and computational requirements remain insufficiently characterised.

Objective:

To benchmark locally hosted open-weight LLMs (1.5–141 billion parameters) for title and abstract screening and structured data extraction in a One Health AMR systematic review, assessing performance, computational efficiency and suitability for human-assisted workflows.

Methods:

The corpus comprised 5,535 records (2020-2025) from a PRISMA-guided review of gene-based AMR biomarkers. Two reviewers screened all records independently, resolving disagreements by consensus. Fourteen 4-bit quantised models, served with Ollama on institutional GPUs without fine-tuning and with one fixed prompt per task, screened every record. Thirteen extracted six fields (pathogen species, sampling location, sampling year, resistance genes, One Health classification and genotypic detection method) from the 3,305 records included at the first human screening stage, scored against dual-reviewer reference extraction. Screening was evaluated with recall, precision, F1-score and accuracy. Extraction was scored per field with token-set fuzzy similarity, Jaccard similarity or exact accuracy and summarised as an unweighted macro-average (0-100), retaining records without a usable response.

Results:

The five leading screening models (24-120 billion parameters) reached F1-scores of 0.83-0.85 and accuracy of 0.76-0.78, compared with 0.78 and 0.63 for an include-all baseline. They retained 93%-97% of eligible records but excluded only 40%-51% of ineligible ones. Extraction macro-averages ranged from 44.3 (deepseek-r1:1.5B) to 90.5 (nemotron-3-super:120b-a12b), with seven models of 4-120 billion parameters within 2.2 points (88.3-90.5). Year of sampling was the best-extracted field (maximum 99.3) and resistance genes the weakest (maximum 85.1). Between 0 and 242 extraction responses per model were unusable. The three leading 24-32B screening models used 9.1-11.1 GPU-hours, against 22.9-35.1 for the two leading 120B models.

Conclusions:

Locally hosted open-weight LLMs demonstrated promising performance for human-assisted systematic review workflows, but important limitations remain. Screening models achieved high recall at the expense of specificity, limiting their ability to reduce manual screening workloads without risking missed eligible studies. Autonomous screening is therefore not supported by these findings, while potential use as a second screener warrants prospective evaluation. For structured extraction, a 4B model performed within 2.2 points of the leading 120B model, although resistance gene extraction remained challenging. Model size alone was not a reliable indicator of performance, with 24–32B models achieving screening F1-scores comparable to 120B models at less than half the GPU-hours. These findings support further evaluation of locally deployed LLMs as computationally efficient, institutionally controlled tools for assisting evidence synthesis.