JMIR Publications

ABSTRACT

Background:

Clinical prediction models suffer from performance drift as patient population shifts over time. There is a great need for model updating approaches or modeling frameworks that can effectively utilize the old and new data.

Objective:

Based on the paradigm of transfer learning, we aimed to develop a novel modeling framework that transfers old knowledge to the new environment for the prediction task, and contributes to performance drift correction.

Methods:

The proposed predictive modeling framework, TransferGBM, maintains a logistic regression-based stacking ensemble of two Gradient Boosting Machine (GBM) models representing old and new knowledge learned from old and new data respectively. The ensemble learning procedure can dynamically balance the old and new knowledge. Using 2010 to 2017 electronic health record data on a retrospective cohort of 141,696 patients, we validated TransferGBM for hospital-acquired acute kidney injury prediction modeling.

Results:

The baseline models (i.e., transported models) that trained upon 2010 and 2011 data suffered from significant performance drift in the temporal validation with 2012 to 2017 data. Refitting these models using update samples would result in performance gain nearly in all cases. The proposed TransferGBM succeeded to achieve uniformly better performance than the refitted models.

Conclusions:

Under the scenario of population shift, incorporating new knowledge while preserving old knowledge will be essential for maintaining stable performance. Transfer learning combined with stacking ensemble learning can help achieve a balance of theses knowledge in a flexible and adaptive way, even in the case of insufficient new data.