JMIR Publications

ABSTRACT

Deploying agentic AI systems without adequate plans for human supervision raises serious concerns about patient safety, privacy, and equity. To address this gap, a group of experts across industry, academic and clinical informatics interested in AI and safety convened a voice AI taskforce to discuss and develop consensus on the impact of agentic AI in healthcare. Through this collaboration, we developed a consensus framework to determine potential risks and plan mitigation efforts based on potential clinical use cases to aid health care delivery organizations assess, implement and evaluate AI agents to meet their needs. Based on five diverse use case examples, we identified common themes of risk at the level of the agent, data, patient and clinician as well as the mitigation strategies needed to address them. Agent-level risks include robust transcription validation, knowledge-grounded responses, mandatory conversation checklists, demographic bias testing, and red-teamed escalation triggers. At the data level, secure identity verification, high‑quality data, interoperable standards and rigorous governance form the foundation of safety. Patient‑level risks include equitable access, patient suitability and clear escalation paths. Finally, clinician-level risks include alert prioritization, defined liability frameworks, workflow-integrated outputs, and preserved clinical override authority. Robust symptom recognition and a thoughtful precision–recall balance are also essential aspects to consider. These guardrails, supported by multidisciplinary oversight and continuous evaluation, can enable AI agents to contribute to patient care without compromising safety, privacy or equity. This framework aims to address the uncertainties in risks to patient safety that should be considered by healthcare delivery organizations to safely apply these technologies to address healthcare needs.