JMIR Publications

ABSTRACT

Background:

Conversational health AI is increasingly being developed for patient-facing applications, but evaluating stochastic, multi-turn systems before human studies remains challenging. Simulated patients could provide repeatable test environments, but their usefulness depends on whether the simulated patient itself reliably preserves the behavioral characteristics defining the test case.

Objective:

We evaluated an evidence-grounded simulated-patient framework for testing conversational health AI, using medication adherence as a behavioral test case. Specifically, we examined whether explicit behavioral representation and structured operationalization improve simulation fidelity beyond the same evidence supplied as prose or conventional patient prompting.

Methods:

Behavioral profiles for South Asian and Chinese simulated patients were derived from independent systematic reviews. Across 80 standardized multi-turn conversations (880 patient responses), we compared structured evidence-grounded simulation with the same evidence supplied as prose, ad hoc prompting, and web-informed patient construction. Automated measures assessed behavioral grounding, recovery of assigned behaviors, fidelity to assigned behavioral characteristics, unsupported barriers, conversation behavior, and correspondence with an independent corpus of published patient utterances.

Results:

Structured operationalization improved fidelity of generated behavior to the intended evidence-derived patient state. In the primary paired ablation, structured operationalization increased mean conversation-level behavioral groundedness compared with the same evidence supplied as prose (0.850 vs 0.732; mean paired difference 0.117, 95% CI 0.077-0.157; Cohen dz=1.376; P<.001). Structure did not improve every measure: prose produced better recovery of assigned COM-B dimensions (F1-score: 0.641 vs 0.503; P<.001), and other outcomes varied by evidence source.

Conclusions:

Evidence-grounded simulated patients provide a computationally evaluable approach for creating repeatable behavioral test cases for conversational health AI. Explicit structure improved behavioral grounding beyond evidence content alone but did not uniformly improve all fidelity measures. This computational stage can support pre-human testing while identifying where subsequent expert and patient validation remains necessary.