JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are increasingly used in clinical decision support, yet it remains unclear whether their outputs can be interpreted as valid risk estimates in clinical prediction.

Objective:

This study aimed to evaluate whether inference-only LLMs can function as probabilistic predictors and to compare their performance with an outcome-trained machine learning (ML) model.

Methods:

We conducted a controlled comparison using identical clinical inputs from ICU patients with spontaneous intracerebral hemorrhage. An outcome-trained XGBoost model was compared with LLM-based predictions generated under multiple prompting strategies. Performance was evaluated in terms of discrimination, decision behavior across probability thresholds, and concordance between SHAP-derived feature importance and LLM-derived feature prioritization.

Results:

The ML model demonstrated superior discriminative performance and stable decision thresholds. In contrast, LLM-based approaches showed substantial decision instability, with wide variation in optimal thresholds across prompting strategies and patterns suggestive of probability misalignment, including high sensitivity and low specificity. Concordance between SHAP-derived attribution and LLM-derived prioritization was modest, indicating divergence between empirical predictive structure and language-based reasoning.

Conclusions:

LLM-generated probabilities may resemble risk estimates but lack stable probabilistic meaning in clinical prediction. These findings suggest that LLM outputs should be interpreted as plausibility-based assessments rather than calibrated risks. A hybrid approach combining LLM-based reasoning with outcome-trained predictive models may provide a more reliable framework for clinical decision support.