JMIR Publications

ABSTRACT

Background:

The Delphi method is widely used to derive expert consensus on complex clinical problems, yet it is slow and resource‑intensive [1]. Recent advances in large language models (LLMs) and retrieval‑augmented generation (RAG) offer the possibility of accelerating consensus while maintaining methodological rigour. LLMs can retrieve and summarise evidence but they frequently hallucinate and cannot reliably cite sources [2]. At the same time, mRNA‑based drugs and vaccines are rapidly moving from concept to clinic, generating a pressing need for timely, evidence‑based consensus on regulatory, manufacturing and clinical issues [3].

Objective:

We evaluated whether a modular, RAG‑enabled, multi‑agent AI pipeline could replicate the post–round 1 behaviour of a human reviewer in a Delphi study. The primary objective was to determine whether AI‑assisted statement revision could rescue a greater proportion of sub‑threshold statements and achieve consensus comparable to human revision by round 2.

Methods:

A parallel, two‑arm Delphi was conducted on 28 statements about RNA medicines. Fifty international panellists (clinicians, researchers and patient representatives) were randomised into human (arm A) and AI‑assisted (arm B) groups. After round 1, statements below the 75 % agreement threshold were revised either manually by human reviewers or by an AI pipeline comprising: (i) a ReferenceDetector to identify external citations, (ii) a Summarizer to produce structured summaries of supporting PDFs, (iii) a hybrid RAG module that combined dense and sparse retrieval with cross‑encoder re‑ranking, and (iv) a Refiner that generated revised statements, reasoning logs and explicit citations. Human experts verified retrieved citations and approved or amended revisions. Agreement rates and vote distributions were compared across arms.

Results:

Arm A reached consensus on 20/28 statements in round 1 (71.4 %), whereas arm B passed 13/28 (46.4 %). After revision, consensus increased to 26/28 statements (92.9 %) in arm A and 24/28 (85.7 %) in arm B. The AI arm exhibited a larger mean improvement (Δ = 39.3 percentage points) because more statements were initially below threshold. Nonetheless, the absolute difference between arms after round 2 was modest (7.2 percentage points). AI‑generated revisions were particularly effective for statements far below threshold, but both arms failed to rescue 2–3 statements owing to substantive disagreements.

Conclusions:

A modular, citation‑anchored AI pipeline can closely approximate human performance in Delphi consensus procedures while substantially reducing manual workload. When paired with human oversight, AI assistance accelerated revision and closed most of the performance gap by the second round. Adoption of AI‑augmented workflows could accelerate consensus development on emerging technologies such as RNA therapeutics, provided that transparency, rigorous retrieval and human review are maintained.