JMIR Publications

ABSTRACT

Background:

Clinical decision support (CDS) tools in electronic health records (EHRs) are often used as core strategies to support quality improvement programs in the clinical setting. Monitoring impact (intended and unintended) of these tools is crucial for program evaluation and adaptation. Existing approaches for monitoring typically rely on healthcare providers’ self-reports or direct observation of clinical workflows, which require substantial data collection efforts and are also prone to reporting bias.

Objective:

To develop a novel monitoring method leveraging EHR activity data and demonstrate its use in monitoring the CDS tools implemented by a tobacco cessation program sponsored by the National Cancer Institute’s Cancer Center Cessation Initiative (C3I).

Methods:

We developed EHR-based metrics to monitor implementation of two CDS tools: (1) a screening-alert reminding clinic staff to complete smoking assessment and (2) a support-alert prompting healthcare providers to discuss support and treatment options, including referral to a cessation clinic. Using EHR activity data, we measured completion (encounter-level alert-completion rate) and burden (the number of times an alert fired before completion and time spent handling the alert) of the CDS tools. We report metrics tracked for 12 months post-implementation, comparing 7 cancer clinics (2 clinics implemented the screening-alert and 5 implemented both alerts) within a C3I Center, and identify factors associated with alert completion.

Results:

The screening-alert fired in 5121 encounters during 12 months post-implementation. The encounter-level alert-completion rate (0.55: clinic staff acknowledged completion of screening in EHR; 0.32: clinic staff completed EHR documentation of screening results) remained stable over time but varied considerably across clinics. On average, screening-alerts fired 2.7 times before completion; time spent completing the alert per encounter was 53 seconds (52 seconds spent postponing alerts per encounter). The support-alert fired in 1074 encounters during 12 months. Providers acted upon (i.e., not postponed) the alert in 87.3% encounters, identified a patient ready to quit and ordered a referral to the cessation clinic in 12.0% and 2.0% encounters. The support-alert fired 2.1 times before completion; time spent completing the alert per encounter was 50 seconds (67 seconds spent postponing alerts per encounter). The screening-alert completion rate was higher in encounters perceived relevant to routine tobacco screening by physicians, comparing with other encounters (63% vs. 3.5%, P<.001). These findings inform four areas where the alert design and use can be improved.

Conclusions:

EHR-activity metrics were able to monitor the success and also the burden of tobacco cessation alerts, allowing a more nuanced understanding of potential tradeoffs associated with alert implementation. These metrics can be used to guide implementation adaptation and are scalable across diverse settings.