JMIR Publications

ABSTRACT

Background:

Cross-corpus multimodal models in mental-health machine learning fuse corpora that share no participants, and are typically validated by a battery of four internal checks: label shuffling, noise injection, a fusion ladder, and expected calibration error.

Objective:

To present a fully documented case in which all four checks passed on a classifier that measured nothing about the construct in its title, to identify the mechanisms responsible, and to propose reporting items that would have detected them.

Methods:

I reanalysed my own bipolar-state classifier (euthymia, depression, mania), built by fusing text and physiological data from four public corpora. The reanalysis preserved full information about how every row and label was created, and added a within-class permutation test, a missingness-mask audit, subject-grouped cross-validation and strictly proper scoring rules. I then rebuilt the physiological corpus from its raw activity series to establish what the data actually support.

Results:

The classifier reached 82.6% macro-F1 and satisfied every check. A provenance-preserving reanalysis accounts for this score through three independent mechanisms, none of which requires genuine information about bipolar state. Rows were paired across modalities by class label and nothing else, so permuting one modality within classes left performance unchanged, at 0.828 against 0.828. The physiological feature vector could not have been measured for any participant in its source dataset, because its two halves exist for disjoint groups of people; the resulting missingness pattern alone predicts the derived label at 0.542 macro-F1 against a chance floor of 0.333. The physiological labels came from a hand-written mapping under which the largest class is undefined time between the conditions of a laboratory protocol. Two further hypotheses were tested and rejected, and are reported as negative results. Rebuilt correctly, the physiological corpus supports a modest result: 0.523 macro-F1 for three-class diagnosis under subject-grouped evaluation, against a chance floor of 0.333.

Conclusions:

The four checks share a blind spot. Each perturbs an assembled feature matrix, and none inspects how that matrix was assembled. I propose a seven-item reporting checklist that closes this gap, and report where its diagnostics returned null in my own analysis. Clinical Trial: Not applicable