JMIR Publications

ABSTRACT

Background:

The pervasiveness of mental health (MH) disorders, and the shortcomings of current diagnoses, such as dependence on patients' subjective recalls and failure to recognize symptoms due to comorbidity, have motivated the exploration of machine learning (ML) in this domain. The algorithms' ability to model correlations between data and subsequently make deductions is promising to support clinical diagnosis. The interactions among data of different modalities also have great potential that is worth investigating.

Objective:

This study systematically reviews existing methodologies that apply ML to multimodal data collected passively from participants to assess features extracted from varying forms of data, fusion techniques adopted to combine them, the subsequent ML algorithms applied to learn from such features, and if any, the findings of the association between modality-specific features and MH disorders. The review focuses on studies adopting passive sensing approaches by hypothesizing that gathering data non-intrusively might be more acceptable and better capture people's natural behaviors.

Methods:

A systematic search was conducted on the Scopus, ACM Digital Library, PubMed, and IEEE Xplore databases to gather studies from January 2015 to May 2022. Studies were screened for eligibility if they gathered passive data involving 2 or more modalities, adopted non-intrusive data collection approaches, and applied ML approaches to explicitly predict the existence or severity of specific MH disorders.

Results:

A total of 9346 studies were obtained from the search results, and 100 were assessed and included in the current review. The primary data source categories were audio and video recordings (n=48), social media (n=30), and smartphone and wearable devices (n=26). The modalities involved were audio, visual, textual, smartphone sensor, wearable sensor, demographics, and personalities. We found that modality-specific features correlate differently with MH disorders in different contexts. The selection of modality fusion techniques from feature, score/decision, and model levels also relies on specific use cases. In addition, determining an effective data-driven ML model, among supervised learning, neural network, ensemble learning, multi-task learning, and others, are dependent on the nature and information contained within the data and how the algorithms learn the potential relationships between data.

Conclusions:

All data sources exhibit great potential in detecting MH disorders, but their qualities in terms of reliability, practicality, and ethicality deserve deeper consideration. Such consideration is significant due to the enormous influence the data has on the value and contribution of research, and the subsequent social impacts that the research outcomes can bring to the MH community.