JMIR Publications

ABSTRACT

Background:

Artificial Intelligence (AI) tools can augment clinical decision making. Clinicians usually interact with AI recommendations through clinical decision support (CDS) systems, but there is limited data on the factors associated with compliance with AI-based CDS.

Objective:

The study aims to evaluate the effectiveness and factors contributing to alert compliance with an artificial intelligence recommendation.

Methods:

The Precision Resuscitation with Crystalloids in Sepsis (PRECISE) trial randomized a subgroup of patients with sepsis to usual care versus an AI-CDS, which recommended balanced crystalloids instead of normal saline. In this secondary analysis of the PRECISE trial, provider and patient-level factors were evaluated for association with alert compliance with chi-squared tests and a mixed-effects logistic regression was performed with random effects.

Results:

1,378 alerts were fired on 876 unique hospitalizations with an overall alert compliance of 40.9% and alert cancellation rate of 26.3%. In unadjusted analysis, the emergency department (ED) had significantly higher compliance (p<0.01, 46.3%, n=1,114) compared to the intensive care unit (ICU) (18.18%, n=264). Sodium ranges (p< 0.01), KDIGO stages (p<0.01), SOFA scores (p<0.01) were all associated with compliance. Compliance rate was significantly higher in physicians (48.2%) compared to Advanced Practice Providers (APPs) (p<0.01). In mixed-effects logistic regression, the ICU had lower odds ratio of compliance compared to the ED (aOR 0.35, 95% CI 0.22-0.56, p<0.1) and APPs had lower odds ratio of compliance compared to physicians (aOR 0.62, 95% CI 0.44-0.89, p<0.01).

Conclusions:

Setting, clinician type, and acuity of the patients all affect alert compliance with AI-CDS. Overall compliance with the trial AI-CDS was similar to previously reported rates in studies of traditional CDS. Further research is needed on methods of improving compliance with AI-CDS.