JMIR Publications

ABSTRACT

Background:

Reliable deployment of machine learning systems in healthcare requires mechanisms for determining whether individual predictions can be trusted. Conventional confidence-based approaches often fail to capture underlying uncertainty, particularly in high-capacity models where predictions may remain highly confident despite unstable reasoning.

Objective:

This study proposes a Decision-Calibrated Explainable AI (DC-XAI) framework for evaluating prediction reliability using stability-based signals derived from both model outputs and feature attribution explanations.

Methods:

The proposed DC-XAI framework integrates two complementary reliability signals: prediction stability under stochastic perturbations and explanation stability measured by feature-attribution consistency. These signals are combined into a three-tier decision system consisting of ACCEPT, ACCEPT WITH CAUTION, and DEFER categories to support reliability-aware clinical decision-making. The framework was evaluated using the MIMIC-IV critical care dataset for in-hospital mortality prediction. Evaluation was conducted using a two-level strategy, comprising a global performance assessment on the full test set (n = 13,074) and a perturbation-based stability analysis on a representative subset (n = 1,000). Logistic Regression, XGBoost, and Multi-Layer Perceptron (MLP) architectures were compared.

Results:

The results revealed a significant Stability–Accuracy Gap across model architectures, demonstrating that predictive performance alone does not reliably reflect prediction trustworthiness. Logistic Regression exhibited a strong monotonic relationship between stability and accuracy, whereas XGBoost demonstrated brittle stability, maintaining stable predictions despite incorrect outputs. The MLP exhibited non-monotonic stability behaviour, where instability in feature attribution did not consistently correspond to prediction failure. These findings indicate that the relationship between stability and reliability is architecture-dependent and that explanation stability alone is insufficient as a universal trust signal.

Conclusions:

The proposed DC-XAI framework provides a practical mechanism for reliability-aware clinical AI deployment by integrating prediction stability and explanation consistency into a triage-based decision process. The findings challenge the assumption that stability is a universal proxy for reliability and highlight the need for architecture-aware trust calibration in safety-critical healthcare AI systems.