JMIR Publications

ABSTRACT

Background:

Healthcare settings serving older adults often encounter challenges in rapidly translating qualitative feedback into program improvements, with the resource-intensive nature of traditional qualitative methods serving as a significant barrier. Large language models (LLMs) offer potential to streamline qualitative data analysis for quality improvement. However, existing research on LLM-based qualitative methods primarily focuses on accelerating or augmenting the work of experienced qualitative researchers, which may not be applicable to lower-resource settings where collaboration with qualitative researchers is infeasible. Furthermore, there is limited evidence on the applicability of LLM-based qualitative analysis to programs serving older adults, people with dementia, and caregivers. This gap hinders efforts to identify scalable solutions to support programs under increasing strain to meet the needs of a growing population of older adults.

Objective:

This study examines the strengths and limitations of LLM-based thematic analysis in healthcare contexts where qualitative research experts are unavailable, using open-ended survey responses from caregivers participating in the Care Ecosystem dementia care program. Additionally, it proposes a workflow for human-LLM collaboration tailored to such settings.

Methods:

Caregiver participants (N=603) in a dementia care intervention provided survey feedback on what was most helpful and what could be improved. Responses were analyzed using two methods: traditional thematic analysis conducted by qualitative researchers and LLM-based thematic analysis using GPT-4o, an LLM produced by OpenAI. LLM outputs were evaluated for alignment with traditionally identified themes, ability to capture meaningful insights in the data, and production of relevant and accurate illustrative quotations. GPT-4o was accessed through a UCSF enterprise platform that enables HIPAA-compliant use of LLMs.

Results:

LLM themes were mostly aligned with traditionally identified themes and captured actionable feedback in the dataset, although two LLM themes misinterpreted caregiver feedback and one traditionally identified theme was not captured. The LLM correctly identified support responding to emergencies as a key benefit of the program, a theme missed by qualitative researchers. Challenges included the fabrication of illustrative quotations not found in the dataset, although only 3.5% of all LLM-produced quotations misrepresented caregiver feedback.

Conclusions:

The LLM’s rapid synthesis of survey responses highlights LLMs’ potential to facilitate timely qualitative analysis for program improvement in healthcare settings serving older adults, such as dementia care programs. To ensure accuracy, we propose a workflow for human-LLM collaboration involving staff review of data prior to LLM analysis. Careful integration of LLMs is needed to promote accuracy, protect participant privacy, and ensure alignment with patient and caregiver priorities.