JMIR Publications

ABSTRACT

Background:

Qualitative analysis helps interpret complex human experiences by identifying patterns not easily captured through quantitative measures. However, when applied at scale in healthcare and health professions education, qualitative coding is a labor-intensive task. As a result, large volumes of narrative data are often underutilized or inefficiently analyzed. Recent advances in artificial intelligence, particularly large language models (LLMs), have introduced new opportunities for scaling qualitative analysis through automated processing of narrative text. Prior work has demonstrated that LLMs can approximate human judgments across qualitative and evaluative applications, including thematic analysis and rubric-based assessment. Still, methodological questions remain regarding how these models assign meaning to narrative data and translate it into structured outputs, particularly when mapping responses onto ordinal Likert-type scales. It also remains unclear how prompt design influences the validity, reliability, and stability of LLM-generated qualitative coding.

Objective:

This article explores an aspect of AI coding with multiple applications in both research and assessment. Specifically, how effectively can AI interpret Likert-type inferences relative to a human.

Methods:

This study addresses these gaps by evaluating whether an LLM (Gemini) can infer ordinal Likert-type responses from narrative data. Using a dataset of paired Likert responses and narrative elaborations, multiple prompt conditions were systematically varied to assess their impact on distributional alignment and interrater reliability.

Results:

Across all prompt conditions, Gemini-generated outputs closely approximated the overall distribution of the original self-reported Likert responses while outperforming a human qualitative analyst in distributional alignment. Agreement at the individual response level remained moderate (κw between 0.44 and 0.52), and variations in prompt structure did not produce substantial changes in performance (Krippendorff’s α ranged from 0.69 to 0.76).

Conclusions:

These findings suggest that LLMs may infer latent meaning structures from narrative data and reproduce aggregate response patterns from an underlying distribution of human responses. LLMs may therefore serve as scalable tools for supporting qualitative coding of ordinal data in healthcare settings when combined with clear processing instructions, structured outputs, and human verification.