JMIR Publications

ABSTRACT

Background:

The evolution of language models, particularly Large Language Models (LLMs), has introduced transformative potential for psychological assessment, challenging traditional rating scale methods that have dominated clinical practice for over a century.

Objective:

This study aimed to develop and validate an Automated Assessment Paradigm (AAP) that integrates natural language processing (NLP) with conventional measurement tools to assess depressive symptoms, exploring its feasibility as a novel approach in psychological evaluation.

Methods:

A cohort of 115 participants, including 28 individuals diagnosed with depression based on the Diagnosis and Treatment of Mental Disorders Guidelines, completed the BDI-Fast Screen (BDI-FS) via a custom ChatGPT interface and the Chinese version of the PHQ-9. Statistical analyses included Spearman’s correlation (PHQ-9 vs. BDI-FS-GPT scores), Cohen’s kappa (diagnostic agreement), AUC evaluation, and logistic regression models comparing BDI-FS (cutoff = 3) and PHQ-9 (cutoff = 5).

Results:

Spearman's analysis revealed a significant correlation between PHQ-9 and BDI-FS-GPT total scores. Cohen’s kappa indicated substantial diagnostic agreement (κ = 0.75, P < .001). The BDI-FS demonstrated high diagnostic accuracy (AUC = 95.3%), detecting 89.3% of depressed patients with an 11.5% false-positive rate. Logistic regression models showed superior performance for BDI-FS compared to PHQ-9 in classification efficiency.

Conclusions:

The AAP framework combines the interactivity and personalization of NLP-powered tools with the psychometric rigor of traditional scales, suggesting a promising paradigm for future psychological assessment. Its ability to enhance engagement while maintaining reliability and validity supports further validation in large-scale studies as LLM technology advances.