Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
AI as the Interpreter: Deciphering Mental Health Stories in the Social Media Realm
ABSTRACT
Background:
Mental health issues, compounded by stigma and lack of understanding, are on the rise globally. This increase turns social media into a valuable yet underutilised resource for early detection and intervention. While advancements in machine learning (ML) and natural language processing (NLP) have progressed, the full potential of AI in identify mental health issues from online narratives remains unexplored.
Objective:
This study analyses two Artificial Intelligence (AI) models, Automated Machine Learning (AutoML) and PaLM 2 (Pathways Language Model). These models are used to identify mental health root causes from social media posts and to align with human judgment. The aim is to explore AI-human correlation and AI interpretability of mental health discussions by measuring its performance to enhance its understanding of mental health issues.
Methods:
We fine-tune AutoML and PaLM 2 on social media datasets and compare their performances against human judgment. This study incorporates quantitative analyses to assess correlation, interpretability, and error patterns, alongside qualitative analysis to evaluate AI's handling of emotional nuances using a subset of (n=50) posts.
Results:
AutoML exhibits high precision and recall at 86%. However, it struggles with overfitting, evidenced by its lower concordance rate of 68%. In contrast, PaLM 2 scores lower on precision and recall at 79% but achieves an 80% alignment rate with human judgment. These findings suggest that PaLM 2 has a better ability to accurately predict mental health issues.
Conclusions:
AutoML's tendency to overfit limits its practical utility. On the other hand, PaLM 2 shows a more balanced performance across different mental health narratives. However, both models require further refinement to more effectively mimic human decision-making. Notably, human raters also encounter difficulties in areas where AI struggles. This emphasises the need for continuous advancements in AI to better comprehend linguistic and emotional contexts of mental health narratives.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.