Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Oct 14, 2019
Date Accepted: Mar 11, 2021
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
A/B testing using rapid randomized controlled trials to optimize clinical decision support
ABSTRACT
Background:
Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to understand and iteratively improve design choices embedded within CDS tools.
Objective:
This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS tools to improve their usability, reduce alert fatigue and promote quality of care.
Methods:
A multi-step process combining tools from user-centered design, A/B testing and implementation science is used to understand, ideate, prototype, test, analyze and improve each candidate CDS. CDS engagement metrics (alert views, ignores, orders) are used to evaluate which CDS version is superior.
Results:
Two experiments are highlighted to demonstrate the impact of the process. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care in a rapid RCT. The new alert text resulted in minimal impact but the failure triggered another round of testing that identified key issues and led to a 70% reduction in alert volume in the next round. In the second experiment, the process was used to test three versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as three supporting images. Three rounds of RCTs showed that the financial framing was 5-10% more effective than the other two but that adding images did not have a positive impact.
Conclusions:
These data support the potential for this new process to rapidly develop, deploy and improve CDS within an EHR. This approach may be an important tool for improving the impact and experience of CDS. Clinical Trial: Our flu alert trial was registered in January 2018 with ClinicalTrials.gov, registration number NCT03415425. Our tobacco alert trial was registered in October 2018 with ClinicalTrials.gov, registration number NCT03714191.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.