JMIR Publications

ABSTRACT

Background:

Clinical pathways (CPW) turn evidence-based guidelines into step-by-step care protocols. When formalized as process models, they can support automated clinical decision support and conformance checking of care trajectories. Large language models (LLMs) can extract process models from text at scale, but their output still falls short of clinician-validated reference models. This gap stems in part from missing context: CPW are written for experts and often omit background knowledge such as task conflicts, dependencies, and priorities. These omissions can compromise fidelity and patient safety in case of LLM-based extraction.

Objective:

This study aimed to develop and evaluate a knowledge enhancement approach for extracting process models from clinical pathways. In this approach, missing background knowledge is formalized as declarative constraints and injected into LLM prompts.

Methods:

We formalized background knowledge as constraints in Linear Temporal Logic over finite traces and as ontological relations. The constraints covered task conflicts, decisional dependencies, prioritized conditional tasks, alternative and part-of tasks, and pathway entry and exit points. In a knowledge enhancement approach, constraints were narrated in natural language and injected into the prompt alongside process modelling guidelines. Conformance of the output with the constraints was then verified by model checking. Using GPT 5.6, we extracted process models from 8 clinical pathways in Ontario’s Quality-Based Procedures repository. An ablation study compared 3 prompt conditions (baseline, + modelling guidelines, + knowledge constraints), each run 5 times per pathway, for 120 extractions in total. Outputs were evaluated by structural similarity to reference models validated by 2 emergency physicians, and by constraint conformance. Conditions were compared on per-pathway means using a Friedman test with exact, Holm-corrected Wilcoxon signed-rank tests.

Results:

Mean structural similarity rose from 0.60 (baseline) to 0.76 with modelling guidelines and 0.84 with knowledge constraints (absolute gain, 0.24; 40.2% relative increase). Prompt condition had a significant effect (χ_2^2=16.0, P<.001), and both the guidelines and the knowledge constraints gave significant stepwise improvements (P=.02). Every pathway improved at each step. Constraint conformance improved to 0.98 after knowledge injection (P=.008). All constraints were met for 7 of 8 pathways; in the community-acquired pneumonia pathway, two constraints conflicted.

Conclusions:

Injecting formalized background knowledge into prompts is a practical, model-agnostic way to improve the fidelity of LLM-extracted clinical pathway models and to guard against unsafe task sequences. Formal constraints allow a feedback loop, where constraints are applied to the input and their conformance checked in the output; non-conformance could be used further as input for a next iteration. Highly complex guideline text and conflicting constraints remain challenges. Future work will distill constraints automatically from public sources, inject them based on relevance to the CPW, and involve large-scale evaluations.