Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Investigating Human Examiner Variability in Objective Structured Clinical Examinations Using Large Language Models as a Context-Invariant Comparator: An Observational Study
ABSTRACT
Background:
Despite extensive standardization, objective structured clinical examination (OSCE) scoring remains subject to examiner-related variability. This variability may reflect individual examiner characteristics, examination context, or candidate-related factors, but their respective contributions remain difficult to quantify. Large language models (LLMs) offer a novel approach by providing a comparator unaffected by examination timing and candidate sequence.
Objective:
To investigate associations between examiner characteristics, individual and pair-level variability, examination context, and OSCE scoring differences using LLM scoring as a context-invariant comparator.
Methods:
In this prospective observational study, fifth- and sixth-year medical students completed summative OSCEs independently scored by two human examiners. Encounters were audio-recorded, transcribed, and scored by ChatGPT GPT-5.5 Thinking and Gemini 3.1 Pro using standardized checklists and a zero-shot prompt. Human-LLM clinical score differences on a 20-point scale were analyzed using mixed-effects models to examine examiner characteristics, individual and pair-associated variability, time of day, preceding candidate performance, and candidate performance level. ChatGPT was the primary comparator, with Gemini used for sensitivity analyses. Internal LLM repeatability was assessed through six independent evaluations of 24 transcripts.
Results:
The study included 381 students contributing 975 encounters across six stations, yielding 1950 human ratings from 54 examiners organized into 30 pairs. Female examiners awarded scores 0.44 points higher relative to ChatGPT than male examiners (95% CI 0.04-0.84), while nonclinical examiners scored 0.64 points higher than medical examiners (95% CI 0.17-1.10). No association was observed with surgical specialty or examiner seniority. Examiner-level deviations from station averages ranged from −1.28 to +1.86 points, with greater estimated variability at the examiner-pair level than at the examiner-specific level (SD 0.79 vs 0.32 points). No consistent association was identified with within-block examination time (P=.37), preceding candidate performance (P=.53), or candidate performance level (P=.84). Examiner-level differences were highly consistent across the two LLM comparators. Across six independent assessments, ChatGPT reproduced 93.8% of clinical encounter-item ratings identically (Fleiss κ=0.934), compared with 90.1% for Gemini (κ=0.916).
Conclusions:
OSCE scoring appeared relatively robust to contextual and candidate-related influences, although variability persisted across individual examiners and examiner pairs. Using LLM scoring as a context-invariant comparator provides a novel way to investigate examiner-related variability.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.