JMIR Publications

ABSTRACT

Background:

LLM-Assisted Content Analysis (LACA) is a modification of traditional content analysis, leveraging the Large Language Model (LLM) to co-develop codebooks and automatically assign thematic codes to online reviews dataset.

Objective:

This study aims to develop and validate the use of LACA for analyzing hospital online reviews, and to identify themes of issues from online reviews using this method.

Methods:

Online reviews for 53 private hospitals in Selangor Malaysia were acquired. Fake reviews were filtered out using natural language processing and machine learning algorithms trained on yelp.com validated datasets. gpt-4o-mini model API was then applied to filter out reviews without any quality issue. 200 of the remaining reviews were randomly extracted and fed into gpt-4o-mini model API to produce a codebook validated through parallel human-LLM coding to establish inter-rater reliability. The codebook was then used to code (label) all reviews in the dataset. The thematic codes were then summarized into themes using factor analysis to increase interpretability.

Results:

14,938 online reviews were acquired in which 1,121 were fake, 1,279 contain negative sentiments (12%) and 9,635 do not contain any negative sentiment (88%). gpt-4o-mini model subsequently inducted 41 thematic codes together with their definitions. Average Human-GPT inter-rater reliability is perfect (kappa = 0.81). Factor analysis identified six interpretable latent factors: 'Service & Communication Effectiveness’, 'Clinical Care & Patient Experience’, ‘Facilities & Amenities Quality’, ‘Appointment & Patient Flow’, ‘Financial & Insurance Management’ and 'Patient Rights & Accessibility'. The cumulative explained variance for the six factors is 0.74 and Cronbach's alpha is between 0.88 - 0.97 (good - excellent) for all factors except factor 6 (0.61 - questionable). The factors identified follow a global pattern of issues identified from literature.

Conclusions:

A data collection and processing pipeline consisting of Python Selenium, the gpt-4o-mini model API, and a factor analysis module can support valid and reliable thematic analysis. Despite the potential for collection and information bias in online reviews, LACA of online reviews is cost-effective, time-efficient, and can be performed in real time, helping hospital managers develop hypotheses for further investigations promptly.