JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are increasingly used to quantify psychological constructs from natural language, yet their outputs are typically evaluated only as predictions of an external label. When that label is itself imperfect, optimizing prediction conflates label agreement with construct measurement. No study has applied a multi-domain measurement-oriented evaluation to LLM-derived psychological outputs.

Objective:

We applied five psychometric evaluation domains to LLM-derived outputs to test whether measurement-oriented evaluation surfaces properties that prediction-oriented evaluation does not: extraction repeatability, convergent validity with a lexicon-based tool, ecological concurrent validity decomposed into within-person and between-person components, incremental value in a four-arm comparison with that tool, and method-blinded expert evaluation with structured error attribution.

Methods:

Thirteen linguistic features were extracted by GPT-5.2 from 3,556 person-days of daily voice narratives collected from 279 call center workers over 15 days via ecological momentary assessment. Features were evaluated across the five domains, with LIWC-22 as the lexicon-based comparator. Multi-model robustness was assessed by re-extracting all features with Gemini 2.5 Flash.

Results:

Features showed high repeatability (mean ICC = 0.987), construct-dependent convergent validity (Spearman rho = 0.08 to 0.79), and within-person tracking of same-day stress (negative emotion beta = 0.68). LLM features increased explained variance in daily stress from 13.2% to 30.4%, versus 16.9% for the lexicon-based comparator. Experts preferred LLM-derived profiles in 83.7% of 312 blinded decisions. Error analysis showed that 53% of false-positive attributions reflected reference-label limitations, not model failures. Re-extraction with Gemini 2.5 Flash replicated key properties (person-level r = 0.94). Three phenomena surfaced only under measurement-oriented evaluation: divergence between within-person and between-person effects, reference-label limitations as a dominant error source, and moderation of feature meaning by individual differences.

Conclusions:

Prediction metrics alone are insufficient for evaluating LLM-derived psychological measures. Measurement-oriented evaluation revealed properties, including level-dependent effects and asymmetric error attribution, that have direct implications for how LLM-based assessment studies interpret their results.