Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Accepted for/Published in: JMIR Research Protocols

Date Submitted: Aug 17, 2023
Open Peer Review Period: Aug 16, 2023 - Aug 18, 2023
Date Accepted: Sep 18, 2023
(closed for review but you can still tweet)

The final, peer-reviewed published version of this preprint can be found here:

Objectively Quantifying Pediatric Psychiatric Severity Using Artificial Intelligence, Voice Recognition Technology, and Universal Emotions: Pilot Study for Artificial Intelligence-Enabled Innovation to Address Youth Mental Health Crisis

Alemu DY, Caulley DD, Sezgin E, Abebe G, Cárdenas Bautista DE

Objectively Quantifying Pediatric Psychiatric Severity Using Artificial Intelligence, Voice Recognition Technology, and Universal Emotions: Pilot Study for Artificial Intelligence-Enabled Innovation to Address Youth Mental Health Crisis

JMIR Res Protoc 2023;12:e51912

DOI: 10.2196/51912

PMID: 37870890

PMCID: 10628686

Objectively Quantifying Pediatric Psychiatric Severity Using AI, Voice Recognition Technology, and Universal Emotions: AI-Enabled Innovation to Address Youth Mental Health Crisis

  • Dr. Yared Alemu; 
  • Dr. Desmond Caulley; 
  • Emre Sezgin; 
  • Girmaw Abebe; 
  • Dr. Elizabeth Cárdenas Bautista

ABSTRACT

Background:

Providing Psychotherapy, particularly for youth, is a pressing challenge in the healthcare system. Traditional methods are resource-intensive, and there is a need for objective benchmarks to guide therapeutic interventions. Automated emotion detection from speech, using artificial intelligence (AI), presents an emerging approach to address these challenges. Speech can carry vital information about emotional states, which can be utilized to improve mental healthcare services, especially when the person is suffering.

Objective:

This study aims to develop and evaluate automated methods for detecting the intensity of emotions (anger, fear, sadness, and happiness) in audio recordings of patients’ speech. We also demonstrate the viability of deploying the models. Our model was validated in a previous publication by Alemu et al. (2023) with limited voice samples. This follow-up study used significantly more voice samples to validate the previous model.

Methods:

We used audio recordings of patients, specifically children with high Adverse Childhood Experience ACE scores; the average ACE score was five or higher, at the highest risk for chronic disease and social/emotional problems; only 1 in 6 have a score of 4 or above. The patients' structured voice sample was collected by reading a fixed script. Four highly trained therapists classified audio segments based on a scoring process of four emotions and their intensity levels for each of the four different emotions. We experimented with various preprocessing methods, including denoising, voice-activity detection, and diarization. Additionally, we explored various model architectures, including convolutional neural networks (CNNs) and transformers. We trained emotion-specific transformer-based models and a generalized CNN-based model to predict emotion intensities.

Results:

The emotion-specific transformer-based model achieved a test-set precision and recall of 86% and 79%, respectively, for binary emotional intensity classification (high or low). In contrast, the CNN-based model, generalized to predict the intensity of four different emotions, achieved test-set precision and recall of 83% for each.

Conclusions:

Automated emotion detection from patients' speech using AI models is found to be feasible, leading to a high level of accuracy. The transformer-based model exhibited better performance in emotion-specific detection, while the CNN-based model showed promise in generalized emotion detection. These models can serve as valuable decision-support tools for pediatricians and mental health providers to triage youth to appropriate levels of mental healthcare services.


 Citation

Please cite as:

Alemu DY, Caulley DD, Sezgin E, Abebe G, Cárdenas Bautista DE

Objectively Quantifying Pediatric Psychiatric Severity Using Artificial Intelligence, Voice Recognition Technology, and Universal Emotions: Pilot Study for Artificial Intelligence-Enabled Innovation to Address Youth Mental Health Crisis

JMIR Res Protoc 2023;12:e51912

DOI: 10.2196/51912

PMID: 37870890

PMCID: 10628686

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.