JMIR Publications

ABSTRACT

Background:

Clinical practice guidelines (CPGs) translate research evidence into actionable recommendations, but their development faces mounting challenges. With dozens of trials and systematic reviews published daily, the traditional 18-24-month development cycle struggles to incorporate emerging evidence, and manual screening, synthesis, and quality assessment consume thousands of person-hours per guideline. The resulting knowledge translation pipeline is slow, inconsistent, and quickly outdated.

Objective:

This review aimed to chart the computational methods empirically tested for automating CPG development, the tasks they address, and the measures used to evaluate them, and to map where that evidence is concentrated and where it is absent for living guidelines.

Methods:

The review followed the Joanna Briggs Institute scoping review framework [10] and PRISMA-ScR reporting guidance [62]. We searched PubMed/MEDLINE and Scopus for studies published between January 2001 and December 2025, supplemented by backward and forward citation searching. Studies were eligible if the automated component acted on a guideline development task, reported original research with empirical testing, and gave sufficient methodological detail. Two reviewers independently screened records and extracted data. Findings were charted across four pillars: retrieval, extraction, encoding, evaluation.

Results:

Forty-seven studies met eligibility criteria, spanning 2001 to 2025, with a marked increase after 2015 and a surge in LLM work in 2024-2025. Pillar counts (non-exclusive): retrieval 6, extraction 15, encoding 26, evaluation 17. In the Retrieval pillar, machine learning classifiers achieved 70%-80% precision and cut screening burden by 20%-50% at high recall, while semantic search tools traded recall for specificity (98%). In the Extraction pillar, rule-based natural language processing achieved an F-measure of 81.96% for structured clinical actions and 71.75% for patient states, and hybrid ontology methods covered 86% of eligibility criteria; large language models (LLMs) performed well on structured fields but degraded on interpretive content. In the Encoding pillar, approaches progressed from formalisms such as GLIF3 and GEM toward FHIR-based representation, semi-automated tools cut encoding time by about 40%, and logic verification detected inconsistencies and coverage gaps missed by human reviewers. In the Evaluation pillar, LLM appraisers preserved the ranking of guideline quality but assigned higher AGREE II scores than human appraisers, by a mean difference of 12.5% in one study (95% limits of agreement -30.6% to 55.5%; ICC 0.753) and by 22 to 60 percentage points across domains in another.

Conclusions:

Automation demonstrates clear promise for bounded, well-structured tasks within guideline development, while robust evidence for end-to-end, production-scale living guideline pipelines remains limited. No included study demonstrated a validated pipeline connecting all four pillars in an operating guideline program, and evidence on clinical safety, workflow integration, and cost-effectiveness is largely absent. Realizing living guidelines will require interoperability standards such as FHIR, and evaluation in real guideline production rather than retrospective benchmarks.