JMIR Publications

ABSTRACT

Background:

Accurately predicting adverse outcomes in acutely ill older patients enables targeted interventions and appropriate care. However, existing screening instruments based on conventional statistical methods have yielded suboptimal prediction performance.

Objective:

We tested whether machine learning algorithms outperform conventional logistic regression in combination with clinical reasoning in predicting adverse outcome in older Emergency Department (ED) patients.

Methods:

The Acutely Presenting Older People (APOP) study included 2,627 older patients presenting to four Dutch EDs. We collected demographics, disease severity, and geriatric assessments during ED visits and followed patients for 90-day mortality or functional decline. The original conventional statistical approach (published in 2028) included selection of predictors by multidisciplinary experts from geriatric assessments, followed by logistic regression, including backward selection. We replicated the original logistic regression model (7 features) and developed decision tree, random forest, and XGBoost models with 7 and 29 features using leave-one-hospital-out cross-validation with nested hyperparameter tuning. We assessed discrimination (AUC), calibration (calibration slope) and feature importance (SHAP).

Results:

Mean age was 78 years; 805 of 2,627 (31%) patients reached the composite outcome of death or functional decline. Compared to the original logistic regression with 7 features (AUC 0.718), machine learning models showed no significant improvement in discrimination: decision tree 0.680 (Δ -0.038, p<0.001), random forest 0.714 (Δ -0.004, p=0.20), XGBoost 0.712 (Δ -0.006, p=0.12). Extended 29-feature models performed similarly, with best performance for XGBoost (AUC 0.728, Δ +0.010, p=0.09). Calibration slopes revealed severe overfitting for single decision trees (slope 0.45–0.55) but acceptable performance for other models (slope 0.86–1.14). SHAP analysis confirmed that clinically selected features in the original model dominated XGBoost importance rankings.

Conclusions:

Machine learning may not outperform simpler models for prediction of poor 90-day outcomes in older ED patients. Conventional approaches with expert input in feature selection were as effective as more data-driven approaches.