JMIR Publications

ABSTRACT

Background:

Clinical referral triage requires clinicians to interpret heterogeneous free-text information and translate it into an operational priority. Large language models (LLMs) may support this task, but their value in health systems depends not only on text interpretation but also on producing structured, auditable outputs that can be incorporated into existing workflows.

Objective:

This formative study aimed to design and evaluate the technical feasibility of a LangChain-based pipeline that extracts clinically relevant information from unstructured urology referrals, applies predefined prioritisation rules, and generates machine-readable classifications suitable for downstream robotic process automation (RPA).

Methods:

A design science research approach was used to develop a proof-of-concept artefact combining LangChain, GPT-4o-mini, rule-informed prompt templates, structured output parsing, and an RPA-ready output schema. Eighteen synthetic urology referral documents were used for formative evaluation. Five members of the development team independently applied the same clinician-derived prioritisation rubric and resolved disagreements by consensus to establish a provisional reference classification. Exact agreement between the AI output and consensus classification was calculated, ordinal agreement was explored using quadratic weighted Cohen κ, and a confusion matrix was used to examine classification disagreements. An independent consultant urologist subsequently reviewed all 18 referrals and assessed the clinical realism and representativeness of the evaluation corpus. No real patient information was used.

Results:

The prototype generated a priority classification and structured rationale for all 18 synthetic referrals. Fifteen of 18 classifications matched the development-team consensus, corresponding to 83.3% exact agreement. The exploratory quadratic weighted Cohen κ was 0.644 (P<.01). The 3 disagreements comprised one Priority 2 referral classified as Priority 1, one Priority 3 referral classified as Priority 1, and one Not Accepted referral classified as Priority 4. Independent specialist review identified differences in the intended classifications of 6 of 18 documents, two duplicated clinical scenarios, and limitations in the clinical representativeness of the synthetic corpus. The system generated machine-readable outputs suitable for downstream automation.

Conclusions:

The proof of concept demonstrates the technical feasibility of connecting LLM-based interpretation of unstructured referrals with rule-informed prioritisation and structured outputs for workflow automation. The findings are formative rather than evidence of clinical performance. Independent specialist review further demonstrated that evaluation quality depends on the validity of the test corpus as well as model behaviour. Future clinical validation should use a prospectively defined, deduplicated evaluation set separated from development, independent specialist assessment, a version-controlled model and prompt configuration, and a prespecified analysis plan. Clinical Trial: Not applicable.