JMIR Publications

ABSTRACT

Background:

The adoption of predictive algorithms in healthcare comes with the potential for algorithmic bias, which could exacerbate existing disparities. Fairness metrics were proposed to measure algorithm bias, but the application to real-world tasks is limited.

Objective:

This study aims to evaluate the algorithm bias associated with the application of common 30-day hospital readmission models and assesses the usefulness and interpretability of selected fairness metrics.

Methods:

This retrospective study used 10.6 million adult inpatient discharges from Maryland and Florida from 2016-2019. Models predicting 30-day hospital readmissions were evaluated: LACE Index, HOSPITAL score, and the CMS readmission measure, which was applied “as-is” (using existing coefficients) and “retrained” (recalibrated with 50% of the data). Predictive performances and bias measures were evaluated. Bias measures included the parity of false negative rate (FNR), false positive rate (FPR), zero-one-loss, and generalized entropy index.

Results:

The retrained CMS model and HOSPITAL score demonstrated the best overall predictive performance and the lowest racial and income bias in both states. In both states, white and higher income patient groups showed a higher FNR while Black and low-income patient groups resulted in a higher FPR and higher zero-one-loss. When stratified by hospital and population composition, these models demonstrated heterogenous algorithm bias.

Conclusions:

Caution must be taken when interpreting measures’ face value. A higher FNR or FPR could potentially reflect missed opportunities or wasted resources, but these measures could also reflect healthcare utilization patterns and gaps in care. The imperfections of health data, analytic frameworks, and the underlying health systems must be carefully considered. Fairness metrics can serve as a routine assessment to detect disparate model performances but are insufficient to inform mechanisms. Such assessment, however, is an important first step toward data-driven improvement to address existing health disparities.