JMIR Publications

ABSTRACT

Background:

In the era of artificial intelligence (AI), event prediction models are abundant. However, considering the limitation of the electronic medical record-based model, including the temporally skewed prediction and the record itself, these models could be delayed or errored.

Objective:

In this study, we developed multiple event prediction models in intensive care units to overcome the temporal skewness of the model and evaluated the robustness against delayed and errored input.

Methods:

A total of 21,738 patients were included in the development cohort. Three events — death, sepsis, and acute kidney injury — were predicted. To overcome the temporal skewness, we build three models for each event, which predict the events in advance of three different times. Additionally, to evaluate the robustness against error and delayed input, we added simulated errors and delayed input and calculated changes in the Area Under the Receiver Operating Characteristic Curve (AUROC).

Results:

The AUROC of each model outperformed those of the conventional scores and the other machine learning models as well as the logistic regression and XGBoost models. In particular, the mortality prediction model for predicting events 3, 6, and 12 hours in advance showed AUROCs of 0.985, 0.982, and 0.995, respectively. In the error input experiment, in addition to our model, other models show a decreased AUROC when the noise increased. In the delayed input experiments, the AUROCs only slightly decreased; however, in comparison to our model, the standard deviations of the AUROCs were higher in other models.

Conclusions:

For a prediction model that was applicable in the real world, we considered not only performance but also temporal skewness, delayed input and input error.