JMIR Publications

ABSTRACT

Background:

After COVID-19 pandemic, the conflict between limited mental healthcare resources and the rapidly growing number of patients is becoming more pronounced. It is necessary for psychologists to borrow Artificial Intelligence (AI) and Natural Language Processing (NLP) methods to analyse the effectiveness of treatment and then increase treatment efficiency.

Objective:

Our aim is to explore the potential of AI-based NLP methods in the mental healthcare domain, by identifying highly accurate and transferable models for predicting the effectiveness of psychotherapy using the textual reviews of patients.

Methods:

We extracted 24 main mental disorders defined by WHO with 41,851 reviews from a large public data set of 16,950 physical and mental illnesses categories with 161,297 reviews. We formulated the problem mathematically and proposed the Unified Interchangeable Model Fusion (UIMF) to construct 6 fused models using the state-of-the-art BERT model, SVM, and RF models and proposed a new loss function to overcome the overfitting and data imbalance problems. To evaluate the performance of those models fairly, we fine-tuned each model and compared their average performance comprehensively in terms of F1-scores, accuracy, Kappa coefficients and training time.

Results:

The Transformer bidirectional encoder + RF model outperformed the state-of-the-art BERT model and other fused models and it became the best model for solving our problem. The average F1-score was 0.880, the accuracy was 0.881, Kappa coefficient was 0.81, and the training time was 38,574.84 seconds. This model is more suitable for users requiring the best performance with sufficient computing resources. The word embedding encoder + RF model whose training time is around 28 times less than the Transformer bidirectional encoder + RF model also showed relatively good performance with an average F1-score of 0.826, an accuracy of 0.832, and Kappa coefficient of 0.72. It can be deployed in environments with limited computing resources.

Conclusions:

We conclude that different NLP models can be used in different deployment conditions, and we offer potential solutions for clinical situations with varying computing resources. The findings can serve as the evidence to support that the NLP methods can effectively assist psychologists to evaluate the effectiveness of their treatment programs and provide precise and standardized solutions. The UIMF of NLP models has the potential to make the field of mental healthcare take a giant step forward.