JMIR Publications

ABSTRACT

Background:

Clinical practice guidelines are intended to translate scientific evidence into actionable recommendations. However, in many domains, particularly in rare or complex diseases, multiple guidelines co-exist and frequently offer conflicting advice. These discrepancies can impede decision-making, undermine consistency in care, and generate confusion for clinicians and policy-makers alike. Existing strategies for resolving such divergences are often labor-intensive, opaque, and insufficiently reproducible across different contexts.

Objective:

This study aimed to develop and validate a reproducible, transferable framework that uses large language model (LLM)-assisted guideline harmonization and a structured Delphi process to systematically resolve conflicting or missing clinical recommendations and generate expert-vetted consensus items.

Methods:

We present a comprehensive methodology that integrates LLMs with a structured, expert-led Delphi process to identify, evaluate, and reconcile conflicting guideline recommendations. The pipeline comprises multimodal document processing, semantic unification of extracted recommendations, and transformation into clearly defined, votable consensus items. Recommendations are assessed for consistency across guidelines, and those lacking clear alignment are subjected to an iterative Delphi process, informed by expert judgment and LLM-synthesized rationales. In areas where all guidelines remain silent or vague, recent landmark studies are identified and analyzed using the same framework to generate additional consensus candidates.

Results:

Applied to the field of penile cancer as a high-stakes use case, this approach yielded a harmonized and validated set of recommendations with clearly documented guideline provenance and item-level Delphi consensus scores. A multi-model LLM panel was employed to cross-check and refine initial outputs, enhancing robustness and minimizing bias in stance classification. The final deliverable is a reproducible matrix of clinically relevant statements, suitable for adoption by professional societies and guideline committees.

Conclusions:

This methodology offers a scalable, transparent, and transferable solution for the systematic resolution of inter-guideline inconsistencies. By combining AI-powered analysis with expert consensus, it enables reproducible harmonization of clinical recommendations and the structured identification of white areas in need of further clarification. Beyond methodological innovation, this framework directly supports implementation research by enabling cross-context harmonization of evidence, reducing uncertainty in practice, and fostering transparency in guideline development.