JMIR Publications

ABSTRACT

Background:

Predictive models play a critical role in clinical decision-making, though unequal outcome frequencies often compromise performance. Evaluation of these models frequently relies on the area under the receiver operating characteristic curve (AUROC), which may not reflect their real-world clinical utility.

Objective:

To apply a structured evaluation of predictive model performance, in a real-world clinical setting, to examine how metric selection influences clinical interpretation in imbalanced scenarios.

Methods:

Retrospective study. Two predictive models using XGBoost were developed to predict kidney replacement therapy (KRT) and mortality, using data from 17,018 hospitalized COVID-19 patients. Performance was evaluated through AUROC, macro-F1, and per-class precision and recall. Rebalancing strategies (random over and under sampling, SMOTE and Tomek links) were tested.

Results:

KRT was performed in 9.5%, and mortality was 18.0%. Although AUROC were high (0.927 for KRT and 0.945 for mortality), macro-F1 scores were lower (0.688 and 0.830, respectively). Per-class metrics for less frequent outcomes were lower: for KRT, precision 0.527 and recall 0.356; for death, precision 0.725 and recall 0.718. Rebalancing had minor effects on the results.

Conclusions:

Discussion: Despite excellent AUROC values, recall and precision for minority classes were poor, underscoring the risk of missed diagnoses and unnecessary interventions. Rebalancing strategies yielded only marginal improvements. In healthcare, where predictive errors have direct consequences for patients and resource allocation, complementary measures such as macro-F1 and per-class metrics are indispensable to ensure robust and clinically meaningful model evaluation. Conclusion: Although AUROC remains the most widely reported metric in predictive modeling, it is insufficient to evaluate prediction models in imbalanced scenarios, even with rebalancing. Complementary measures such as macro-F1 and per-class metrics should be systematically reported to avoid misleading interpretations.