JMIR Publications

ABSTRACT

Background:

While large language models (LLMs) demonstrate remarkable linguistic fluency, it remains unclear whether they truly understand the people behind the words.

Objective:

This study investigates whether LLMs can infer individual’s big five personality profiles from semi-structured interviews paired with validated self-report big five scores derived from the BFI-10, providing an ecologically valid benchmark that reflects authentic human communication.

Methods:

We systematically compare three modeling paradigms: (1) zero-shot and chain-of-thought prompting with GPT-4.1 Mini, (2) parameter-efficient fine-tuning (LoRA) applied to RoBERTa and Meta-LLaMA architectures. and (3) regression models using static embedding from pretrained BERT and OpenAI’s text-embedding-3-small.

Results:

Across paradigms, correlations with self-reported scores were modest (average r<.26). Chain-of-thought reasoning produced minimal improvement over zero-shot inference; while embedding-based and LoRA-tuned models achieved slightly higher but still sub-reliable alignment, consistent with reliability ceiling of the short-form BFI-10 inventory (α ≈ .65–.75).

Conclusions:

These findings suggest that, despite strong linguistic fluency, current LLMs capture only limited aspects of human personality as expressed in real-world dialogue. By linking natural conversation with validated psychological measures, this work provides an empirically grounded benchmark for advancing psychologically aligned and personalized AI systems.