JMIR Publications

ABSTRACT

Background:

Patient-facing conversational artificial intelligence (AI) agents are increasingly deployed across health care to sustain two-way patient communication and collect patient-reported data at scale. Their workflows include a classification layer that reads each free-text message and routes it to an appropriate response. Performing this triage with a large language model (LLM) is accurate but carries recurring cost and latency at scale and sends sensitive patient text to an external service; a smaller, locally hosted model is cheap, private, and controllable but is less reliable on ambiguous or out-of-scope (out-of-distribution) messages and can make clinically consequential errors. F1 metrics compound the problem by treating every error as equivalent, while in this context misclassifying a safety-critical message is far more harmful than confusing two benign categories. We evaluate triage errors using an asymmetric clinical cost matrix.

Objective:

We aimed to develop and retrospectively evaluate a two-stage triage system that preserves LLM-level quality while reducing how often the LLM is called. We evaluate it by clinical consequence rather than by raw error count.

Methods:

A compact local classifier handled routine messages, and an LLM was invoked only when accepting the local prediction carried high expected clinical cost. The routing rule is the system’s central design choice: each message is escalated by its expected clinical cost under an asymmetric, expert-elicited cost matrix, rather than by generic confidence or uncertainty. The local classifier is a fine-tuned text-embedding encoder trained with outlier exposure and a geometry-aware output head (a Semantic-Anchor Equiangular Tight Frame, SA-ETF, that aligns maximally separated class directions to each category’s clinical description). We studied two programs at one academic health system: 1) a hypertension remote-monitoring program, and 2) a type 2 diabetes self-management study, with 21 and 39 in-scope categories plus a Beyond Scope category. In both programs, the analyzed messages were authored by researchers and staff testers exercising the assistant. Five experts curated the cost matrix, penalizing clinically consequential undertriage more heavily than benign misclassification or overtriage. Gold-standard test sets comprised 114 hypertension and 95 diabetes messages that were independently labeled and adjudicated.

Results:

Across both settings the hybrid matched the clinical cost of LLM-only triage while escalating only 57%-59% of messages. On hypertension its balanced accuracy exceeded LLM-only by a small, consistent margin (+0.026 across seeds; significant in the pooled paired bootstrap, p<0.05); on diabetes the two were comparable. Routing by expected cost also escalated 97%-100% of out-of-scope messages to the LLM instead of letting the local model silently assign them an in-scope label, and it reduced LLM use and external message transmission by 41%-43%.

Conclusions:

Selective, clinically cost-aware escalation can preserve LLM-level triage quality while reducing operational and data-governance burdens. Because the evaluation was retrospective, this is a development-stage evaluation; prospective validation is needed before deployment.