JMIR Publications

ABSTRACT

Background:

Accurate clinical outcome prediction using Electronic Health Records (EHRs) is crucial for patient care and resource allocation. EHRs include both structured data and rich, unstructured clinical notes. However, prior machine learning methods struggle with the multi-modality, long context of notes, and severe class imbalance in clinical tasks.

Objective:

To introduce and evaluate KAMELEON (Knowledge-Augmented Multimodal EHR LEarning for Outcome predictioN), a unified, two-stage hybrid framework that integrates diverse EHR modalities and external biomedical knowledge to enhance clinical risk prediction

Methods:

This study used the publicly available, de-identified MIMIC-III dataset, which includes structured and unstructured data for over 40,000 Intensive Care Unit (ICU) patients. The two tasks studied were 30-day readmission (approximately 4% positive rate) and in-hospital mortality prediction (approximately 13% positive rate). Train-test splits were patient-disjoint (80:20). Performance was evaluated against general and medical Large Language Models (LLMs) and structured baselines. Key metrics included the Area Under the Receiver Operating Characteristic curve (AUROC), Area Under the Precision-Recall Curve (AUPRC), and Macro F1-score.

Results:

The KAMELEON framework consistently outperformed all existing baselines. • 30-Day Readmission. The KAMELEON-BalancedRF model achieved an AUROC of 0.845 and a Sensitivity (Recall) of 0.79. Ablation analysis demonstrated the critical role of the LLM-generated reasoning, with its removal causing the AUROC to drop from 0.844 to 0.7 and sensitivity to fall by over 80%. • In-Hospital Mortality: The KAMELEON-XGBoost model achieved an AUROC of 0.92 and an AUPRC of 0.650. Unstructured-only models showed limited ability to discern mortality, with AUROC values near chance (around 0.51–0.53).

Conclusions:

KAMELEON is the first systematic framework to enhance LLMs for healthcare prediction through graph-guided knowledge retrieval combined with structured machine learning. The framework demonstrates superior performance across both prediction tasks, highlighting the synergistic value of combining diverse data modalities and LLM reasoning for robust clinical risk estimation.