JMIR Publications

ABSTRACT

Background:

Suicide-related outcomes in the United States show pronounced and diverging racial disparities, yet computational models that predict suicide-related hospitalization risk from digital behavioral data have rarely been evaluated for whether they generalize equitably across racial groups.

Objective:

We evaluated whether classical machine learning classifiers trained on a majority-race (White) cohort's Google search queries generalize, in terms of standard classification metrics, to participants from minority racial groups across six retrospective time windows preceding a suicide-related psychiatric hospitalization.

Methods:

We combined three previously collected cohorts pairing Google search history with psychiatric hospitalization records (66 participants; 889,503 queries). Support vector machine (SVM) and k-nearest neighbors (KNN) classifiers were trained via 3-fold cross-validation on White participants' GloVe-embedded query windows and evaluated, without retraining, on a held-out cohort of participants from minority racial groups, across windows of 1 week to 1 year prior to hospitalization.

Results:

KNN performance favored the White cohort over the non-White cohort in every window (weighted F1 differences of 0.01–0.14). SVM's weighted-F1 gap was smaller and inconsistent in direction, converging toward parity by 1 year; disaggregating by class showed this partly reflected a specificity advantage for the non-White cohort (up to 0.92 vs. 0.79 at 1 year) alongside comparable sensitivity. In a smaller, single-cohort subsample, a long short-term memory model achieved 0.97 accuracy on White participants and 0.92 on non-White participants.

Conclusions:

Racial disparity in search-query-based suicide-hospitalization prediction was evident but inconsistent across model classes, underscoring the need to evaluate algorithmic fairness using multiple modeling approaches rather than a single classifier.