JMIR Publications

ABSTRACT

Background:

There is limited analysis into text messaging-based data in mHealth intervention apps. Natural language processing (NLP), through topic modeling, can be used to achieve this, grouping textually and semantically similar texts into coherent topics in large datasets.

Objective:

To explore using BERTopic, an NLP tool, in combination with qualitative methods to identify representative topics and analyze trends through topic modeling messages exchanged between patients and patient navigators in NextStep, an HIV patient support app designed to facilitate HIV linkage and retention (LAR) in care.

Methods:

We analyzed messages exchanged between English-speaking patients and patient navigators in the NextStep app between February 2020 and August 2022. The messages were preprocessed and de-identified before topic modeling using two BERTopic models, one for patient messages and one for navigator messages. Then, two coders condensed the generated topics into distinct thematic categories. Messages not sorted into topics by the model were qualitatively coded for analysis. Data validation was performed by calculating the CV coherence score for each model and Cohen Kappa coefficient for inter-rater agreement between the models and the qualitative coders. The proportion of patient and navigator-initiated topics per patient were compared generally, by patient use-time tertiles, and by endline HIV virologic suppression.

Results:

A total of 2,337 patient messages and 3,264 patient navigator messages were analyzed. Per patient, the most prevalent topics found in patient messages was App Set-Up (33.1%, 95% CI 28.2%-38.0%). Additionally, Appointment Help Request (11.9%, 95% CI 9.3%-14.4%) was distinctly prevalent when compared to other topics implicated in direct HIV care. The most frequent topics for patient navigator messages per patient were Appointment Assistance (21.0%, 95% CI 18.4%-23.7%) and Conversation (16.9%, 95% CI 14.4%-19.5%). Across patient messages stratified by use-time tertiles, the App Set-Up topic decreased in prevalence with increasing use-time, while the Help Request topic increased in prevalence. Within navigator messages, the Introduction topic similarly decreased in prevalence by patients’ use-time tertiles, and the Medication Assistance topic increased in prevalence from the short-term use tertile to mid-term or long-term use tertiles. When stratified by patient study endline HIV viral load, no differences in prevalence were found for either patient or patient navigator message topics.

Conclusions:

BERTopic is a highly promising tool for integrating user-generated content into the design and evaluation process for messaging mHealth interventions. The high prevalence of the App Set-Up topic reflects a high drop off patient usage after first use, and the prevalence of appointment-related topics in relation to other care topics suggest that the NextStep messaging feature was used primarily to set up appointments for HIV care-related tasks. These use patterns suggest that future LAR messaging interventions should emphasize focus on appointment scheduling features and initial app retention in their design process.