JMIR Publications

ABSTRACT

Recent advancements in Large Language Models (LLMs) have sparked interdisciplinary interest in their ability to assess psychological constructs, particularly Personality. While prior machine learning research has focused on evaluating LLMs’ capability to infer personality traits, often via zero-shot or few-shot learning, few studies have systematically examined the applicability of LLM embeddings for Personality Prediction within a robust psychometric validity framework or explored their correlation with psychological and linguistic features. Addressing this gap, we investigate performance of LLM embeddings on a well-labeled PANDORA dataset (Big Five Personality traits from Reddit). Our key contributions are: (1) demonstrating that LLM embeddings significantly outperform zero-shot approaches in personality prediction; (2) validating the reliability of embeddings using psychometric evaluation and analyzing their correlations with linguistic features (e.g., LIWC, emotional markers); (3) contrasting embedding performance against advanced feature engineering methods (joint and independent use of psycholinguistic features); and (4) revealing that model size influences efficacy, OpenAI outperforming RoBERTa. Our findings underscore the potential of LLM embeddings as a scalable and interpretable tool for computational personality assessment.