JMIR Publications

ABSTRACT

Background:

Computerized physician order entry (CPOE) systems are incorporated with decision support systems (CDSS) to reduce medication errors and improve patient safety. An automatic alert generated from CDSS to directly assist physicians in taking useful clinical decisions, and shaping prescribing behavior. Multiple studies reported that approximately 90–96% alerts are overridden by physicians which actually raises questions about the effectiveness of CDSS. There is intense interest in developing sophisticated methods to combat alert fatigue, but there is no consensus on the optimal approaches so far.

Objective:

To develop machine learning prediction models for reducing the alert fatigue of disease-medication related CDSS.

Methods:

We collected data from a disease-medication related CDSS from a university teaching hospital in Taiwan. We considered prescriptions that triggered alerts in the CDSS between August 2018 and May 2019. Machine learning prediction models such as an Artificial Neural Network (ANN), Random Forest (RF), Naïve Bayes (NB), Gradient Boosting (GB) and Support Vector Machine (SVM) were used to develop prediction models. The data were randomly split into training (80%) and testing (20%) data sets.

Results:

A total of 6,453 prescriptions were used in our model. The ANN machine learning prediction model demonstrated excellent discrimination (area under the receiver operating curve = 0.94 and accuracy = 0.85) whereas RF, NV, GB, and SVM had AUROC of 0.93, 0.91, 0.91 and 0.80, respectively. The sensitivity and specificity of the ANN model were 0.87, 0.83.

Conclusions:

In this study, ANN showed substantial performance in predicting individual physician feedback of an alert from disease-medication CDSS. To our knowledge, this is the first study to use machine learning models to predict physician feedback of an alert; it, however, can help to develop sophisticated CDSS in real-world clinical settings. Clinical Trial: N/A