JMIR Publications

ABSTRACT

Background:

Racial disparities in healthcare are well documented in the United States. As machine learning methods become more common in healthcare settings, it is important to ensure that these methods do not contribute to racial disparities through biased predictions or differential accuracy across racial groups.

Objective:

To assess a machine learning algorithm (MLA) that was intentionally developed to minimize bias in in-hospital mortality predictions between white and non-white patient groups.

Methods:

Bias was minimized through preprocessing of algorithm training data. We performed a retrospective analysis of electronic health record data from patients admitted to the intensive care unit at a large academic health center between 2001 and 2012, drawing data from the MIMIC-III database. Patients were included if they had at least ten hours of available measurements after intensive care unit (ICU) admission, had at least one of every measurement used for model prediction, and had recorded race/ethnicity data. Bias was assessed through the equal opportunity difference (EOD). Model performance in terms of bias and accuracy was compared with the Modified Early Warning Score (MEWS), the Simplified Acute Physiology Score (SAPS)-II, and the Acute Physiologic Assessment and Chronic Health Evaluation (APACHE).

Results:

The MLA was found to be more accurate than all comparators, with a higher sensitivity, specificity, and area under the receiver operating characteristic (AUROC). The MLA was found to be unbiased (EOD = 0.016, P = .204). APACHE was also found to be unbiased (EOD = 0.019, P = .107), while SAPS-II and MEWS were found to have significant bias (EOD = 0.038, P = .006 and EOD = 0.074, P < .001, respectively).

Conclusions:

This study indicates there may be significant racial bias in commonly used severity scoring systems, and that machine learning algorithms may be able to reduce bias while improving on the accuracy of these methods.