Currently submitted to: JMIR Formative Research
Date Submitted: Jun 3, 2026
Open Peer Review Period: Jun 4, 2026 - Jul 30, 2026
(currently open for review)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Using generative artificial intelligence to build trustworthy point of care decision support for clinicians: a mixed methods approach to the development of Dyna AI
ABSTRACT
Background:
Generative artificial intelligence (GenAI) tools can rapidly synthesize clinical information, but require evaluation for accuracy, safety, and bias. Few retrieval‑augmented generation (RAG)-based clinical decision support (CDS) systems have been evaluated in real‑world clinical settings.
Objective:
To describe the development of Dyna AI, a RAG‑based CDS system powered by DynaMed content, and evaluate its usability, equity, and safety.
Methods:
We used an iterative, user-centered process to develop and assess Dyna AI. Phase 1 included beta testers and independent testers who provided survey responses and flagged outputs for clinical review. A clinical assessment team evaluated potentially unsafe responses, and equity experts performed targeted testing. Phase 2 examined real‑world use among DynaMed customers with access to the commercial beta version.
Results:
In Phase 1, 357 testers conducted 15,590 searches and submitted 3,085 instances of feedback. Among 105 survey respondents, 94% reported they would likely use Dyna AI in practice. Safety review, which targeted the highest-risk subset of user-flagged queries, identified 130 potentially harmful outputs (0.7% of all queries), which were addressed through editorial or technical mitigation where needed. Fewer than 0.01% of searches were flagged for equity concerns; equity experts found the responses accurate, with minor opportunities for clearer language. In Phase 2, 12,601 clinicians used Dyna AI, with only 11 users (0.1%) disabling the AI capability to revert to the traditional product experience.
Conclusions:
Dyna AI demonstrated high usability, minimal safety concerns, and low rates of flagged bias in both structured testing and real‑world use. These findings suggest that carefully governed RAG‑based systems can support rapid, trustworthy clinical decision-making when deployed with appropriate clinical oversight and continuous quality review. Ongoing monitoring and iterative refinement will be essential as GenAI in healthcare continues to evolve.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.