JMIR Publications

ABSTRACT

Background:

In the pediatric intensive care unit (PICU), quantifying illness severity can be guided by risk models to enable timely identification and appropriate intervention. Logistic regression models, including the Pediatric Index of Mortality 2 (PIM-2) and Pediatric Risk of Mortality III (PRISM-III), produce a mortality risk score using data that is routinely available at PICU admission. Artificial neural networks (ANNs) outperform regression models in some medical fields.

Objective:

In light of this potential, we aimed to examine ANN performance, compared with logistic regression, for mortality risk estimation in the PICU.

Methods:

The analyzed dataset included patients from North American PICUs whose discharge diagnostic codes indicated evidence of infection, and included the data used for the PIM-2 and PRISM-III calculations and their corresponding scores. We stratified the dataset into training and test sets, with approximately equal mortality rates, in an effort to replicate real-world data. Data pre-processing included imputing missing data through simple substitution and normalizing data into binary variables using PRISM-III thresholds. A two-layer ANN model was built to predict pediatric mortality, along with a simple logistic regression model for comparison. Both the developed models used the same features required by PIM-2 and PRISM-III. Alternative ANN models using a single-layer or unnormalized data were also evaluated. Model performance was compared using area under the receiver operating characteristic curve (AUROC) and the area under the precision recall curve (AUPRC) and their empirical 95% confidence intervals (CIs).

Results:

Data from 102,945 patients (including 4,068 deaths) were included in the analysis. The highest-performing ANN (AUROC 0.871, 95% CI 0.862-0.880; AUPRC 0.372, 95% CI 0.345-0.396) that used normalized data performed better than PIM-2 (AUROC 0.805, 95% CI 0.801-0.816; AUPRC 0.234, 95% CI 0.213-0.255) and PRISM-III (AUROC 0.844, 95% CI 0.841-0.855; AUPRC 0.348, 95% CI 0.322-0.367). The performance of this ANN was also statistically significantly better than the logistic regression model (AUROC 0.862, 95% CI 0.852-0.872; AUPRC 0.329, 95% CI 0.304-0.351). The performance of the ANN that used unnormalized data (AUROC 0.865, 95%CI 0.856 to 0.874) was slightly inferior to our highest-performing ANN; the single layer ANN architecture performed poorly and was not investigated further.

Conclusions:

A simple ANN model performed slightly better than the benchmark PIM-2, PRISM-III scores, and a traditional logistic regression model trained on the same dataset. The small performance gains achieved by this two-layer ANN model may not offer clinically significant improvement, but suggests further research with other or more sophisticated model designs, and better imputation of missing data, may be warranted.