JMIR Publications

ABSTRACT

Background:

Objective Structured Clinical Examinations (OSCEs) are central to medical education assessments, yet human examiner bias linked to both examiner and candidate demographics have been reported. Large Language Models (LLMs) are proposed as scalable alternatives for clinical communication assessments but whether they introduce or reduce demographic bias is not known.

Objective:

The aim was to determine whether three commercial LLMs; GPT-5.2 (OpenAI), Gemini-3-pro-preview (Google) and Grok-4 (xAI), exhibit systematic demographic bias when scoring clinical communication transcripts. Secondary aims included characterising and explaining inter-model differences.

Methods:

An OSCE transcript from the Member of the Royal College of GPs (MRCGP) case handbook was adapted to a borderline pass level. Each model scored the consultation across a full factorial design, manipulating the ethnicity (White Scottish, Pakistani, Other African) and Sex (male, female) for both the doctor and patient, yielding 36 unique demographic combinations. A control condition where no demographics were mentioned was tested. Each condition was scored 100 times per model (total number of prompts asked per model = 3,700, 11,100 across the 3 models total). Per-model four-way factorial ANOVAs, with post-hoc Dunnett's and Tukey tests quantified bias.

Results:

All three models demonstrated significant demographic bias. This bias varied in direction and magnitude. GPT-5.2 had the largest proportion of its score variance attributable to demographic factors (27.8%), then Gemini-3-pro-preview (10.1%), then Grok-4 (1.9%). Patient demographics (particularly patient ethnicity) drove greater score variance than doctor demographics across all models. The Pakistani female patient condition was associated with the lowest mean doctor score across all three models. Main effects were small, but specific intersectional combinations reached very large effect sizes (up to d = −2.84).

Conclusions:

LLMs exhibit systematic demographic bias when assessing clinical communication skills, with the greatest bias observed in GPT-5.2. The bias varies in magnitude, direction and mechanism. The model with the least documented bias intervention (Grok-4) produced the least bias, suggesting that intensive safety training (in the two other models) may introduce scoring artifacts rather than eliminate bias. These findings carry implications for the responsible integration of LLMs into high-stakes medical assessment.