JMIR Publications

ABSTRACT

Background:

Conversational AI systems are increasingly deployed in healthcare for clinical decision support, but their performance varies substantially across patient communication styles, health literacy levels, and behavioral patterns. Static benchmarks cannot capture multi-turn dynamics through which this variation compounds, and no current evaluation framework implements structured AI risk management guidance for conversational healthcare AI. The result is a structural risk: AI systems may perform well in aggregate while failing disproportionately for the populations they are intended to help.

Objective:

This study develops and validates a patient simulation framework that aligns with the National Institute of Standards and Technology (NIST) AI Risk Management Framework (AI RMF) MAP and MEASURE functions, providing an empirical basis for identifying and characterizing performance risks in conversational clinical AI across medical, linguistic, and behavioral patient variation. We applied the framework to a conversational decision aid for antidepressant selection in major depressive disorder (the AI Decision Aid).

Methods:

The simulator integrates three profile dimensions: (1) medical profiles constructed from All of Us electronic health records using risk-ratio gating; (2) linguistic profiles modeling a health literacy gradient and condition-specific communication; and (3) behavioral profiles representing cooperative, distracted, and adversarial engagement. We generated 500 simulated conversations and evaluated profile fidelity through human annotation and an LLM judge, then assessed downstream effects on the AI Decision Aid's concept retrieval and antidepressant recommendations.

Results:

The patient simulator expressed medical concepts with high fidelity (96.6% accurate across 8,210 concepts), with human inter-annotator agreement of 0.73 κ and LLM-judge agreement against human annotators of 0.78 κ. Behavioral profiles were reliably distinguished (0.93 κ), and linguistic profiles showed moderate agreement (0.61 κ). The framework revealed monotonic degradation in AI Decision Aid performance across the health literacy gradient. Rank-1 concept retrieval increased from 47.6% for limited health literacy to 81.9% for proficient health literacy, with corresponding declines in antidepressant recommendation accuracy.

Conclusions:

Patient simulation grounded in the NIST AI RMF exposes measurable performance risks in conversational healthcare AI that static benchmarks miss, with direct equity implications. Health literacy operates as a structural risk factor, with degraded performance concentrated in patients carrying the greatest burden of psychiatric illness. The framework supports targeted risk-mitigation interventions before deployment, and the open-source infrastructure generalizes beyond antidepressant selection.