JMIR Publications

ABSTRACT

Background:

Targeting more homogeneous patients is essential in sepsis clinical trials. Estimating the uncertainty of the artificial intelligence (AI) model outputs and identifying uncertain outcomes based on adjustable confidence levels for human review is also important in clinical decision-making.

Objective:

We aimed to design an AI-based model for purposeful patient enrollment, ensuring that a septic patient recruited into a trial would still be persistently ill by the time the proposed therapy could impact patient outcome. We also expected the model could provide interpretable factors and estimate the uncertainty of the model outputs at a customized confidence level.

Methods:

In this retrospective study, 9135 septic patients requiring vasopressor treatment within 24 hours after sepsis onset were enrolled from Beth Israel Deaconess Medical Center. 7308 and 1827 patients were randomly selected as the development and internal validation cohorts, respectively. 3743 septic patients from the eICU Collaborative Research Database were used as the external validation cohort. All included septic patients were stratified based on disease progression trajectories: rapid death, recovery, and persistent ill. A total of 148 variables were selected for predicting the three trajectories. Four machine learning algorithms with three different setups were utilized. We estimated the uncertainty of the model outputs using conformal prediction (CP). The Shapley additive explanation method was used to explain.

Results:

Multiclass gradient boosting was identified as the best-performing model with good discrimination and calibration performance in both validation cohorts. The area under the receiver operating characteristic curves (AUROCs) with 95% CI was 0.911 (0.880-0.938) for rapid death, 0.850 (0.831-0.867) for recovery, and 0.813 (0.793-0.832) for persistent ill in the internal validation cohort. In the external validation cohort, the AUROCs were 0.879 (0.860-0.898) for rapid death, 0.765 (0.750-0.779) for recovery, and 0.701 (0.683-0.717) for persistent ill. The maximum norepinephrine equivalence, total urine output, acute physiology score-III, mean systolic blood pressure and intravenous fluid administrated contributed the most. Compared to the model without CP, using the model with CP at a mixed-confidence approach reduced overall prediction errors by 24.2% and 29.6% in the internal and external validation cohorts, respectively, as well as enabling the identification of more potentially persistent ill patients.

Conclusions:

The proposed model can accurately identify patients with different disease courses. The usage of CP for estimating the uncertainty of the model outputs allows for a more comprehensive understanding of the model's reliability and assists in making informed decisions based on the predicted outcomes. Therefore, the implementation of our model has the potential to reduce heterogeneity and enroll more homogeneous patients in sepsis clinical trials.