JMIR Publications

ABSTRACT

Background:

Prediction models based on machine learning (ML) methods are being increasingly developed and adopted in healthcare. However, these models may be prone to bias and be considered unfair if they demonstrate variable performance in population subgroups. An unfair model is of particular concern in bladder cancer where disparities have been identified in sex and race subgroups.

Objective:

In this study, we developed a ML model to predict survival after radical cystectomy for bladder cancer, and evaluated for potential model bias in sex and race subgroups. We then compared algorithm unfairness mitigation techniques to improve model fairness.

Methods:

We trained and compared various ML classification algorithms to predict 5-year survival after radical cystectomy using the National Cancer Database. The primary model performance metric was the F1 score. The primary metric for model fairness was the equalized odds ratio (eOR). We compared three algorithm unfairness mitigating techniques to improve the eOR.

Results:

We identified 16,481 patients; 23.1% were female; 91.5% were ‘White’, 5.0% were ‘Black’, 2.3% were ‘Hispanic’, and 1.2% were ‘Asian’. The 5-year mortality rate was 74.6%. The best naive model was XGBoost which had an F1 score of 0.860 and eOR of 0.619. All unfairness mitigation techniques increased the eOR, with correlation remover showing the highest increase and resulting in a final eOR of 0.750. This mitigated model had F1 scores of 0.86, 0.904, and 0.824 in the full, Black male, and Asian female test sets, respectively.

Conclusions:

Our ML model predicting survival after radical cystectomy exhibited bias across sex and race subgroups. By employing algorithm unfairness mitigation techniques, we improved algorithmic fairness as measured by the eOR.Our study highlights the role of evaluating for model bias but also actively mitigating such disparities to ensure equitable healthcare delivery. We also deploy the first online fair ML model predicting survival after radical cystectomy at https://nayanlab.shinyapps.io/fair_cystectomy_survival/.