JMIR Publications

ABSTRACT

Background:

Policy institutions increasingly need evidence synthesized in weeks rather than years. Rapid reviews meet that need, but the compressed timeline usually forces a trade-off against thoroughness and rigour. Large language models (LLMs) now perform well on individual review tasks such as screening and data extraction, and a few purpose-built systems have automated large parts of the systematic review workflow. Those systems, however, are custom-built software that most researchers cannot access or reproduce.

Objective:

This paper describes a transferable methodology for conducting rigorous rapid reviews with general-purpose AI agent tools, and reports its screening accuracy against human screening, the human oversight it required, and the effort involved when applied to a real commissioned policy review.

Methods:

ARROW (Agentic Rapid Review Orchestration Workflow) is a methodology in which a human reviewer directs a general-purpose coordinating agent. The coordinating agent divides a rapid review into parallel specialist agents (search, screening, extraction, appraisal, and synthesis) and separate checking agents that try to disprove each extracted claim before it is accepted. The workflow builds in human decision points at protocol, inclusion, and synthesis, keeps a complete record of every step, and is designed to meet emerging reporting standards. We applied it to a PROSPERO-registered rapid review (CRD420261362746) of methods for assessing, adjusting, and integrating administrative and low-coverage survey data for child anthropometric estimation, conducted to inform the 2027 revision of the WHO, UNICEF and World Bank Joint Child Malnutrition Estimates methodology, and checked the agents against a human-screened sample.

Results:

The workflow screened 34,757 identified records through a tiered agent pipeline, using a high-throughput first-pass agent and then three independent screening agents with different instructions under a two-of-three majority rule. Every record the agents included then went to human review. Checked against the human-screened sample (372 records, about 8% of the pool), the pipeline reached a Cohen κ of 0.884, sensitivity of 93.3%, and specificity of 96.5%, against a human-to-human κ of 0.805 on an overlapping sample. Full text was retrieved for 772 of 978 records, and agents extracted 480 studies into a structured template with human checking. Every reported count was recomputed independently, and any figure that could not be audited was labelled as an estimate. The review was completed in about 15 calendar days with an estimated 40 hours of human reviewer time.

Conclusions:

General-purpose AI agent tools make multi-agent evidence synthesis practical for individual researchers and small teams without software engineering resources. Rigour is protected by independent checking agents and by human control at every stage. We discuss what this means for research capacity and methods training, and the conditions under which the approach is appropriate.