JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) could streamline healthcare instrument validation by serving as scalable, systematic expert panels and qualitative researcher surrogates. This is particularly relevant as traditional instrument development is time- and resource-intensive. Currently, there is a significant gap in validated trust instruments for pediatric emergency and surgical contexts. Such tools are key because trust is foundational to the patient family-physician relationship and is associated with improved care-seeking and treatment adherence.

Objective:

This study had two objectives: (1) to develop new trust instruments and conduct their preliminary content validation through a Synthetic Instrument Validation (SIV) approach that integrates human and LLM capabilities, and (2) to evaluate appropriate use cases for LLMs in psychometric assessment.

Methods:

Two new trust instruments were developed, one for patient families and one for physicians. In phase one, the instruments underwent a two-stage content validation process using parallel synthetic and human expert panels (5 patient-family personas, 3 physician expert personas and 7 unique human experts). Synthetic panels consisted of three persona-prompted LLMs (Claude Sonnet 4, GPT-5, Grok 4), with human panels serving as comparators. The Scale-Content Validity Index (S-CVI) and Fleiss' kappa (κ) acceptance thresholds were set at ≥0.80. In phase two, LLM performance in quantitative research tasks was evaluated. The patient family instrument underwent Flesch-Kincaid assessment, and both instruments underwent cosine similarity analyses using three parallel methods: algorithmic, LLM-instructed, and LLM-derived.

Results:

In phase one, human–synthetic expert panels Fleiss' κ values for dimensional validation were 0.84 (95% CI [0.72, 0.96]) for the patient family and 0.87 (95% CI [0.72, 1.00]) for the physician instrument. For contextual validation, κ values were 0.83 (95% CI [0.73, 0.93]) and 0.88 (95% CI [0.80, 0.96]), respectively. All instrument sections met or exceeded S-CVI ≥0.80 thresholds across both rounds in all panels. Phase two Flesch-Kincaid metrics converged across all three methods (grade level 8.1 ± 1.1; readability score 60.1 ± 5.6), meeting accessibility standards and demonstrating methodological similarity. In contrast, cosine similarity analyses revealed significant LLM performance limitations; cosine similarity analyses were therefore abandoned in favor of the algorithmic method alone, which achieved a maximum cosine similarity of 0.83, indicating acceptable item distinctiveness overall.

Conclusions:

Persona-prompted LLMs effectively assisted preliminary subjective psychometric assessments and reduced timelines from months to weeks but showed limitations in quantitative computations. This suggests that LLMs currently excel in qualitative assessments, while falling short in rule-based and deterministic computations. These findings help establish task-dependent boundaries for LLM integration in psychometric research, necessitating selective human-LLM collaboration. While these instruments achieved preliminary content validity, they require full psychometric testing with patient families and physicians before they can be considered validated for use. This hybrid SIV framework accelerates preliminary healthcare instrument development while maintaining validation rigour.