JMIR Publications

ABSTRACT

Background:

Passive smartphone sensing shows promise for suicide prevention, but behavioral metadata (GPS, screen time, accelerometry) often lacks the contextual information needed to detect acute psychological distress. Analyzing what people actually see, read, and type on their phones—rather than just usage patterns—may provide more proximal signals of risk.

Objective:

We tested whether vision-language models (VLMs) applied to passively captured smartphone screenshots can predict momentary suicidal ideation.

Methods:

Seventy-nine adults with past-month suicidal thoughts or behaviors completed ecological momentary assessments (EMA) over 28 days while screenshots were captured every 5 seconds during active phone use. We fine-tuned open-source VLMs (Qwen2.5-VL, LFM2-VL) and text-only models (Qwen3) to predict suicidal ideation from screenshots captured in the 2 hours preceding each EMA. To avoid data leakage, we used within-person temporal holdouts (70/30 split) and between-person subject holdouts (50/50 split).

Results:

The analytic sample comprised 2.5 million screenshots from 70 participants. Within-person models achieved strong discrimination at the EMA level (AUC=0.83, AUPRC=0.77), with image-based models outperforming text-only models (AUC=0.83 vs 0.79; P<.001). Between-person generalization was near chance (AUC≈0.50), though a simple lexical screening method retained modest discrimination (AUC=0.62). Smaller models performed comparably to larger models, supporting feasible on-device deployment.

Conclusions:

Screen content predicts short-term suicidal ideation with clinically meaningful accuracy when models are personalized, but does not generalize across individuals. These findings support a two-stage clinical architecture: coarse lexical screening for new patients, with personalized VLM-based monitoring after a calibration period. On-device inference may enable privacy-preserving deployment.