Currently submitted to: JMIR mHealth and uHealth
Date Submitted: Oct 10, 2025
Open Peer Review Period: Oct 28, 2025 - Dec 23, 2025
(closed for review but you can still tweet)
NOTE: This is an unreviewed Preprint
Warning: This is a unreviewed preprint (What is a preprint?). Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn (a note "no longer under consideration" will appear above).
Peer review me: Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period (in this case, a "Peer Review Me" button to sign up as reviewer is displayed above). All preprints currently open for review are listed here. Outside of the formal open peer-review period we encourage you to tweet about the preprint.
Citation: Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
Final version: If our system detects a final peer-reviewed "version of record" (VoR) published in any journal, a link to that VoR will appear below. Readers are then encourage to cite the VoR instead of this preprint.
Settings: If you are the author, you can login and change the preprint display settings, but the preprint URL/DOI is supposed to be stable and citable, so it should not be removed once posted.
Submit: To post your own preprint, simply submit to any JMIR journal, and choose the appropriate settings to expose your submitted version as preprint.
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Identification of User Issues with Mental Health Apps via Social Listening: A Machine-Assisted Topic Analysis of Social Media Posts
ABSTRACT
Background:
Mobile apps marketed on the basis that they support mental health have become very popular in recent years. Given their popularity, it is crucial to identify any issues that users experience while using such apps. Identifying these issues and how users experience them may provide insight regarding the safety and suitability of these apps for those seeking mental health support.
Objective:
Unlike existing research in this field, in which user experience issues have been identified by researchers through their personal analysis of the apps, the objective of this study was to generate themes relating to user experience issues using comments from the app users themselves. An additional aim was to evaluate a human-in-the-loop machine learning technique - in which structural topic modelling (STM) was employed - for the analysis of vast volumes of data gathered from X (formerly Twitter), the data source for this project.
Methods:
Data relating to 5 of the most popular mental health apps was gathered from the X application programming interface (API) using R (the programming language). An unsupervised topic modelling approach was evaluated, testing models with 5–40 topics and differing covariates. Two human researchers conducted thematic analysis to interpret the topics. A structural topic model with 10 topics – each containing 20 X-posts - was identified as the most appropriate for acquiring insights.
Results:
Using R, 19,603 X-posts were gathered from the X API that related to 5 of the most popular mental health apps. These were posted any time between March 2006 (X / Twitter’s inception) and December 2022. The researchers collaboratively produced labels for each of the 10 topics to articulate the primary user experience issue raised by the 20 comments in each topic. Topic 3 was discarded due to lack of coherence and consistency of the comments that related to user experience of the apps, while topic 5 was discarded as the comments related to the apps’ X account posts and not users’ experiences of the apps themselves. The remaining 8 labelled topics were organised into 4 themes, which highlighted issues some users experienced with the apps: The first theme, guidance shortfall, encompassed difficulties in following the guided meditations, challenges in selecting appropriate material from a broad content library, and incompatibility between app usage and users’ home environments. The second theme, technical difficulties, referred to problems relating to subscription and access, as well as various technical faults experienced within the apps. The third theme, heightened emotions related to app-affiliated celebrities, captured emotional reactions such as over-excitement linked to celebrity involvement, as well as feelings of anger directed towards particular affiliated celebrities. Finally, a standalone theme emerged concerning the negative impact of sleep self-monitoring, whereby some users reported that tracking their sleep via the apps had a detrimental effect on their sleep experience.
Conclusions:
The combination of STM and human qualitative analysis of X comments revealed several issues that users experienced while using popular mental health apps, often resulting in negative outcomes. This study provides further evidence that STM can be combined with human qualitative methods to help rapidly analyse large volumes of social media data and provide insight regarding the user experience of mass reach digital health interventions. Clinical Trial: N/A
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.