JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) have shown promise for automated scoring of objective structured clinical examinations (OSCEs), but evidence across diverse assessment items and authentic clinical encounters remains limited. Whether agreement with human examiners is sufficient to support their replacement, particularly in high-stakes assessments, is uncertain.

Objective:

To evaluate agreement between transcript-based LLM scoring and paired human examiners across summative and formative OSCEs, assess the consequences of replacing human examiners with LLMs, and characterize recurrent mechanisms of disagreement.

Methods:

In this prospective observational study, authentic OSCE encounters were independently scored by two human examiners. Transcripts were assessed by two general-purpose LLMs (GPT-5.0-5.5 Thinking and Gemini 2.5-3.1 Pro) using station-specific checklists and a standardized zero-shot prompt. Agreement was evaluated for binary clinical items and ordinal attitude and communication items. For summative OSCEs, alternative human-LLM scoring configurations were compared with the mean score of both human examiners to assess changes in pass-fail classification and within-station ranking. Clinical-item variants assessed in at least 15 encounters and showing mean LLM-human discordance ≥15% underwent exploratory qualitative analysis.

Results:

The study included 1,423 encounters across 35 stations, comprising 975 summative encounters from six stations and 448 formative encounters from 29 stations. These generated 18,551 clinical-item ratings per rater across 482 distinct clinical-item variants, and 3,376 attitude-item ratings per rater across 14 domains. Exact clinical-item agreement was 94.1% between human examiners, compared with 87.5% for ChatGPT-human and 87.3% for Gemini-human comparisons (Cohen κ=0.88, 0.75, and 0.75, respectively). The mean number of discordant clinical items per encounter was 0.77 between humans versus 1.63-1.65 between LLMs and humans. Exact agreement for attitude and communication items was lower (55.6% human-human versus 37.7%-39.4% LLM-human). In summative OSCEs, pass-fail classifications changed in 5.0% of encounters with a single human examiner, 7.1%-8.6% with human-LLM pairs, and 12.6%-14.5% with LLM-only configurations. Median absolute within-station percentile shifts were 2.5, 5.5, and 8.5-9.7 points, respectively. Among 279 clinical-item variants assessed in at least 15 encounters, 91 (32.6%) met the ≥15% discordance threshold and were classified into seven recurrent mechanisms: unobservable gestures (n=3), two-stage station processes (n=9), composite criteria (n=19), completeness threshold ambiguity (n=13), semantic equivalence (n=22), open versus targeted questioning (n=17), and contextual or clinical inference (n=8).

Conclusions:

General-purpose LLMs achieved high agreement with human examiners for clinical-item scoring but were not interchangeable with them for individual assessment decisions or ranking. These findings support further investigation of LLM-assisted formative assessment while highlighting the need for task-specific adaptation, improved scoring rubrics, and human oversight before implementation in high-stakes summative OSCEs.