JMIR Publications

ABSTRACT

Background:

Machine learning and natural language processing have demonstrated significant potential for mental health assessment: Describing your mental health in your own words can offer a more ecologically valid approach than traditional rating scales. However, very few models go beyond single-label diagnosis or constructs, and even fewer are validated against high-quality reference assessments.

Objective:

This study develops a language-based assessment model that assesses the need for mental health support based on probed natural language and validates it against best-estimate assessments from multiple experienced psychotherapists.

Methods:

We analyzed an enriched online sample (N = 600 for development and N = 212 for validation), in which about half reported experiencing mental health issues. Participants described their mental health using open-ended responses regarding 1) mental health, 2) suicidal thoughts, 3) medical history, and 4) depression. The responses were converted into contextual word embeddings using a large language model (LLM) and entered as predictors in a ridge regression using nested cross-validation. Need for mental health support was assessed on a scale of 1 (No support needed) to 5 (Potential crisis) by two to three experienced psychotherapists. Their assessments were based on longitudinal clinical information and were averaged into a best-estimate assessment for model validation. We used the Sequential Evaluation with Model Preregistration framework, which separates model development from validation in a held-out set to support robust estimations and generalizability.

Results:

The language-based assessments closely aligned with the best-estimate assessments (r=.82) and showed strong correlations with established clinical rating scales for depression (PHQ-9), anxiety (GAD-7), stress (PSS-10), and suicidality (IDAS; r=.62-.77). Language-based visualizations of topics and word embeddings showed that low need for support assessments were associated with mentioning well-being and good health, while high assessments were related to depression, anxiety, and suicidality.

Conclusions:

This study demonstrates that natural language responses analyzed through LLMs and machine learning can be used to assess individual’s need for mental health support in close alignment with best-estimate assessments from experienced psychotherapists. Using less than five minutes of respondent time, this approach offers a practical tool for early-stage mental health screening in both clinical and self-assessment contexts.