Accepted for/Published in: JMIR mHealth and uHealth
Date Submitted: Oct 10, 2025
Open Peer Review Period: Oct 28, 2025 - Dec 23, 2025
Date Accepted: Jun 18, 2026
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Identification of User Issues with Mental Health Apps via Social Listening: A Machine-Assisted Topic Analysis of Social Media Posts
ABSTRACT
Background:
Mobile apps marketed on the basis that they support mental health have become very popular in recent years. Given their popularity, it is crucial to identify any issues that users experience while using such apps. Identifying these issues and how users experience them may provide insight regarding the safety and suitability of these apps for those seeking mental health support.
Objective:
Unlike existing research in this field, in which user experience issues have been identified by researchers through their personal analysis of the apps, the objective of this study was to generate themes relating to user experience issues using comments from the app users themselves. An additional aim was to evaluate a human-in-the-loop machine learning technique - in which structural topic modelling (STM) was employed - for the analysis of vast volumes of data gathered from X (formerly Twitter), the data source for this project.
Methods:
Data relating to 5 of the most popular mental health apps was gathered from the X application programming interface (API) using R (the programming language). An unsupervised topic modelling approach was evaluated, testing models with 5–40 topics and differing covariates. Two human researchers conducted thematic analysis to interpret the topics. A structural topic model with 10 topics – each containing 20 X-posts - was identified as the most appropriate for acquiring insights.
Results:
Using R, 19,603 X-posts were gathered from the X API that related to 5 of the most popular mental health apps. These were posted any time between March 2006 (X / Twitter’s inception) and December 2022. The researchers collaboratively produced labels for each of the 10 topics to articulate the primary user experience issue raised by the 20 comments in each topic. Topic 3 was discarded due to lack of coherence and consistency of the comments that related to user experience of the apps, while topic 5 was discarded as the comments related to the apps’ X account posts and not users’ experiences of the apps themselves. The remaining 8 labelled topics were organised into 4 themes, which highlighted issues some users experienced with the apps: The first theme, guidance shortfall, encompassed difficulties in following the guided meditations, challenges in selecting appropriate material from a broad content library, and incompatibility between app usage and users’ home environments. The second theme, technical difficulties, referred to problems relating to subscription and access, as well as various technical faults experienced within the apps. The third theme, heightened emotions related to app-affiliated celebrities, captured emotional reactions such as over-excitement linked to celebrity involvement, as well as feelings of anger directed towards particular affiliated celebrities. Finally, a standalone theme emerged concerning the negative impact of sleep self-monitoring, whereby some users reported that tracking their sleep via the apps had a detrimental effect on their sleep experience.
Conclusions:
The combination of STM and human qualitative analysis of X comments revealed several issues that users experienced while using popular mental health apps, often resulting in negative outcomes. This study provides further evidence that STM can be combined with human qualitative methods to help rapidly analyse large volumes of social media data and provide insight regarding the user experience of mass reach digital health interventions. Clinical Trial: N/A
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.