Currently submitted to: JMIR AI
Date Submitted: Nov 12, 2026
Open Peer Review Period: Nov 18, 2026 - Jan 13, 2027
(currently open for review)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Deriving an Entropy Threshold from Conformal Prediction at a Matched Review Volume for Patient-Level Uncertainty Flags in Mental Health Treatment Prediction: Development and Evaluation Study
ABSTRACT
Background:
In mental health treatment, often more than one treatment is defensible for a patient. Current machine learning models report deterministic predictions without a per-patient confidence measure, which limits clinical usability. Uncertainty quantification methods measure this confidence and flag uncertain patients for review. Conformal prediction returns a set of plausible treatment classes for each patient, with a coverage guarantee set by a chosen error rate, alpha. Entropy measures uncertainty as the spread of a patient’s predicted probability distribution across the treatment classes and has no generally agreed ‘uncertain’ flagging threshold.
Objective:
This study derives an entropy threshold from conformal prediction at a matched review volume and tests whether both methods select the same patients and capture errors equally, so that both methods can be reported together for each prediction.
Methods:
Data came from 2,525 patients referred to Internetpsykiatrien, a Danish internet-delivered cognitive behavioral therapy service, across four treatment classes. Five models were trained: bootstrap logistic regression and support vector machine ensembles, Bayesian logistic regression, neural network with Monte Carlo dropout and a gradient boosted tree ensemble. Each model ran on two feature representations for a total of ten runs. Conformal prediction sets its threshold at a pre-specified alpha of 0.05, with alpha also evaluated from 0.01 to 1. Patients with empty or multi-label conformal prediction sets are flagged as uncertain. For entropy, the matched review volume threshold is the 1–r quantile of calibration entropy, where r is the share of calibration patients’ that conformal prediction flags as ‘uncertain’. Patients with entropy above this threshold are flagged as ‘uncertain’.
Results:
Balanced accuracy ranged 0.663 to 0.700. At alpha 0.05, the two test flag rates differed by at most 0.040, or 20 of 505 patients. The error recall of both flags exceeded that of a same-size random queue by 0.149 to 0.313. The 95% interval of the two flags paired difference in error recall included zero in 9 of 10 runs. Cohen’s kappa was 0.674 to 0.941. Up to alpha 0.10, median kappa was 0.72 or above, and at alpha 0.20 it fell to 0.24. Spearman’s rho between entropy and the second-largest predicted probability was 0.692 to 0.978.
Conclusions:
The matched review volume threshold method gives entropy a threshold from conformal prediction. Up to alpha 0.10, the two flags select largely the same patients, and the two review queues hold a similar share of the model’s errors. So, both methods can be reported together. At higher alpha, the two review queues stay matched in length but hold increasingly different patients. The two continuous scores order some patients differently. Reporting both gives each prediction a set of plausible classes and a measure of predicted probability distribution across the four treatment classes.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.