JMIR Publications

ABSTRACT

Background:

Depression diagnosis relies on symptomatology and challenges health professionals who depend on subjective evaluations of patients’ reported experiences and observable behavior. Although novel machine learning approaches can objectively quantify speech changes, few studies have systematically compared the longitudinal performance of acoustic and linguistic speech markers in a clinically diagnosed sample.

Objective:

This study aimed to develop between- and within-person classifiers to assess the severity of depression and monitor changes to detect treatment response or recurrence. A secondary objective was to compare different speech modalities in predicting changes in depressive symptoms.

Methods:

We collected 348 voice audio recordings from 104 patients diagnosed with mood disorder over a one-year period. Depression severity was assessed using the Hamilton Depression Rating Scale (HAMD). Acoustic and linguistic features were extracted using the OpenSMILE toolkit and Linguistic Inquiry and Word Count (LIWC) frameworks, following automatic speech recognition and machine translation. Mixed-effect multivariate linear regression was used to evaluate the associations between speech markers and HAMD scores, adjusting for covariates, including age, sex, body mass index, diagnosis, and antipsychotic dosage for acoustic features and age, sex, diagnosis, and years of education for linguistic features. Light Gradient Boosting and eXtreme Gradient Boosting were used as base learner algorithms. We developed between-person classifiers to detect moderate-to-severe depression and within-person classifiers to detect treatment response or relapse and compared their performance across different speech modalities. Hyperparameter tuning and 95% confidence interval estimation were performed using a bootstrap bias-corrected cross-validation approach combined with a grid search. The models were validated using a separate held-out set and fivefold cross-validation.

Results:

We identified significant differences in the acoustic and linguistic speech patterns of patients with depression. Patients with depression showed greater temporal instability in key spectral properties of their speech and diminished pitch variation and used words related to death and negative emotions more frequently than their counterparts. The between-person classifier combining acoustic and linguistic features detected moderate-to-severe depression with an area under the receiver operating characteristic curve (AUC) of 0.78 in a held-out set, compared with the model using demographic features (AUC 0.64). Within-person classifiers detected treatment response above AUC 0.80 and disease relapse above 0.90 across all modalities, surpassing the demographic model (AUC 0.58 and 0.64, respectively).

Conclusions:

Between- and within-person comparisons of speech markers can be leveraged in detecting and monitoring depression. We demonstrate the feasibility of applying LIWC-based psycholinguistic analysis to machine-transcribed and translated speech, supporting the replicability of this approach across languages. Automated multimodal voice analysis can be integrated into digital health platforms, such as telemedicine and smartphone-based monitoring applications, providing a scalable and effective approach for accessing mental health care.