Red Teaming a Patient-Facing Clinical AI Platform: Two-Phase Prospective Study
ABSTRACT
Background:
Health systems are increasingly deploying patient-facing conversational artificial intelligence (AI) platforms, yet validated methodologies for safety evaluation under realistic high-risk conditions are lacking. Prior work has documented safety failures in clinical AI but not shown whether iterative, health system-embedded testing can produce measurable safety improvement.
Objective:
This study aimed to evaluate a structured red teaming methodology for safety assessment of a patient-facing clinical AI platform and to determine whether targeted remediation guided by initial testing reduced safety failures in subsequent rounds of high-risk evaluations.
Methods:
A 2-phase prospective simulation study was conducted at a single academic health system in January and February 2026. Phase 1 enrolled 48 healthcare professionals who completed 6 randomized clinical scenarios (3 medium-risk, 3 high-risk). Following independent platform remediation informed by aggregated Phase 1 findings, Phase 2 enrolled 27 additional participants who completed eight targeted high-risk scenarios. All 476 transcripts were independently scored by 3 calibrated raters using a 5-domain safety rubric. A strict firewall separated the research team from the platform developer, and only aggregated, deidentified findings were shared between phases.
Results:
In Phase 1, the high-risk failure rate was 30.2% (35 of 116 transcripts) versus 5.6% (7 of 126) for medium-risk scenarios. In Phase 2, the high-risk failure rate was 8.5% (20 of 234), representing an absolute difference of 21.7 percentage points (participant-clustered 95% CI, 11.1-32.2) and a 72% relative difference, with comparable performance across pre-specified and improvised prompts within the targeted failure categories. Interrater reliability was moderate to strong in Phase 1 (ICC 0.66, 95% CI 0.62-0.69) and moderate in Phase 2 (ICC 0.49, 95% CI 0.43-0.55). Agreement on the binary failure classification remained stable between phases (Fleiss κ =0.70 in both phases) with observed agreement increasing from 90.9% to 94.9% . Residual failures clustered around triage calibration errors and scope violations.
Conclusions:
Structured iterative red teaming was associated with improved safety performance on new high-risk scenarios following targeted remediation. The 5-domain rubric and scenario taxonomy offer a replicable methodology for health systems conducting pre-deployment and ongoing clinical AI safety assessment. Clinical Trial: This study was not registered as a clinical trial.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.