JMIR Publications

ABSTRACT

Background:

Large language models are increasingly used in health research and clinical education. However, they may exhibit social desirability bias (SDB), a tendency to present overly favourable responses when evaluation is inferred.

Objective:

The objective of this study was to quantify the presence and extent of social desirability bias in large language models across mental health and lifestyle screening instruments by evaluating the impact of contextual awareness on model outputs.

Methods:

Four OpenAI models (GPT-4o, GPT-4o-mini, o1, o1-mini) were tested on four instruments sensitive to SDB: GAD-7, PHQ-9, AUDIT, and FANTASTIC. Each model completed 100 "naïve" trials (single items presented independently) and 100 "informed" trials (full questionnaires, revealing evaluative context). Responses were compared to human normative baselines using z-tests. Differences between naïve and informed responses were assessed using t-tests and reported in units of human standard deviations (SDs).

Results:

All models deviated significantly from human norms on at least two instruments (p<0.001). Informed prompts consistently produced more socially desirable outputs. Mean PHQ-9 and GAD-7 scores dropped by up to 11.9 points (1.3 SD); AUDIT and FANTASTIC scores shifted toward "healthier" profiles. Smaller models showed greater SDB on mental health tools, whereas larger reasoning models showed greater SDB on lifestyle questionnaires.

Conclusions:

Large language models systematically under-report stigmatised symptoms and behaviours when they detect evaluation. While this mimics human patient behaviour and may potentially aid high-fidelity simulation, it may introduce significant systematic error when models are used for objective data processing or synthetic cohort generation.