JMIR Publications

ABSTRACT

Background:

Frontier large language models (LLMs) now pass national health-professions licensing examinations, and such scores are increasingly read as evidence of competence. Whether a high score means the same thing for a model as for a human is a question of construct validity, not leaderboard rank. Spain gates specialty training across four professions through parallel, negatively marked, single-MCQ ranking examinations (medicine [MIR], nursing [EIR], psychology [PIR], pharmacy [FIR]), a post-hoc-released, Spanish-language test bed.

Objective:

This study aimed to determine whether the near-ceiling performance of frontier LLMs on four Spanish specialty-access examinations is psychometrically interchangeable with human examination performance, that is, whether machine item difficulty, error structure, and solution process track those of human candidates, rather than merely whether the models score highly.

Methods:

We assembled 839 valid 2026 items (MIR 203, EIR 220, PIR 210, FIR 206) and evaluated a 12-configuration frontier panel: 6 models (GPT-5.2 at low/medium/high reasoning effort, Claude Opus/Sonnet/Haiku 4.5) crossed with {neutral, expert} prompt frames, each run once. Beyond accuracy we related human facility to machine solvability via classical item analysis, analyzed error and distractor concordance, tested no-knowledge surface solvability, and ran a Claude-family multi-judge audit. Bootstrap 95% CIs resample items only and are lower bounds on total uncertainty.

Results:

These exams increasingly fail to discriminate among the strongest frontier configurations (while still separating weaker models such as Claude Haiku): models near-saturated all four (best pooled 94.8%; AI-panel KR-20 0.86 to 0.95; ensemble oracle 100% on MIR and FIR; 80.7% of items at the ceiling). Machine scores were not psychometrically interchangeable: human and machine difficulty were weakly coupled (Spearman 0.40, 95% CI 0.34 to 0.45; disattenuated shared variance ≈20%). A ceiling-aware simulation settled the ceiling confound. The one-axis null expected ρ=0.63 (predictive interval 0.59 to 0.66), yet the observed 0.40 fell below it (P<.001), robust across 1PL/2PL/3PL and a Δ sweep, indicating divergence beyond a uniform advantage. Models resisted the distractors defining human difficulty (traps: AI 79.5% vs humans 24.4%) and, when erring, hit the dominant human distractor less than a plausibility-matched human (excess −5.1 pp, 95% CI −8.1 to −2.0). Surface leakage existed (34.4%) but was not differentially exploited. Stated certainty discriminated well (GPT-5.2 AUROC 0.87 to 0.90) yet 10 of 12 configurations were mildly net under-confident. By a clinician-unvalidated audit, correct answers were mostly faithful yet predominantly factual recall (reversing on MIR); 6% to 12% of errors were judged clinically dangerous (judge-dependent; κ=0.38).

Conclusions:

In this single Spanish 2026 cohort, near-saturating scores were not psychometrically interchangeable with human scores: these exams increasingly fail to discriminate among the strongest frontier configurations (while still separating weaker models such as Claude Haiku), and machine difficulty does not track human difficulty. Recall-versus-reasoning and danger findings are provisional pending clinician adjudication. This is a single-design case study with implications for assessment redesign.