JMIR Publications

ABSTRACT

Background:

Qualitative thematic analysis is widely used in health research to examine patient experiences and inform the refinement of digital health interventions, but it is time- and labor-intensive. Large language models (LLMs) may help accelerate this process, yet their performance may depend not only on the model itself but also on how the analytic workflow is structured. Current evidence remains limited on how different LLMs perform across multi-stage thematic analysis workflows and across multiple health-related qualitative datasets.

Objective:

To evaluate a modular human–AI collaboration pipeline for LLM-assisted thematic analysis and compare how model choice and workflow strategy influence alignment between AI-generated and human-generated themes across three qualitative health studies.

Methods:

The framework was applied to identify interview transcripts from three completed studies. Three LLMs were compared: Gemini-3-Pro, ChatGPT-5.2-thinking, and Opus-4.6. The workflow separated analysis into code extraction, code combination, and theme generation, and five strategies were tested. AI-generated themes were embedded using sentence-t5-xxl and compared with human-generated themes using cosine similarity after alignment with Hungarian and Greedy matching. Runtime and output-format consistency were also examined.

Results:

Output volume differed substantially by model. Gemini-3-Pro generated the fewest codes and themes, while ChatGPT-5.2-thinking showed a similar but higher output ceiling. Opus-4.6 produced the largest and most variable codebooks and theme sets. Across the three studies, Opus-4.6 showed the strongest and most consistent alignment with human-generated themes, with the best cosine similarity scores observed in POTS-DC (0.893 ± 0.041), COPD-DG (0.891 ± 0.027), and ILD-L3 (0.889 ± 0.032). ChatGPT-5.2-thinking was competitive in selected settings, whereas Gemini-3-Pro generally produced slightly lower similarity scores but had the shortest runtime. ChatGPT-5.2-thinking and Opus-4.6 also showed better formatting consistency and workflow usability than Gemini-3-Pro.

Conclusions:

A modular human–AI pipeline can support thematic analysis across multiple digital health interview studies, but performance depends strongly on both model choice and workflow design. Opus-4.6 produced the most consistently human-aligned themes, while Gemini-3-Pro and ChatGPT-5.2-thinking showed different tradeoffs in speed, fidelity, and usability. These findings support the use of LLMs as structured, human-supervised analytic assistants rather than replacements for qualitative researchers.