JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) generate clinical benchmarks at scale, often validated by requiring agreement among multiple LLM providers. Position bias in individual LLM answers to multiple-choice questions is documented on general-domain and USMLE-style benchmarks, but whether the bias survives multi-provider consensus on EHR-derived medication decisions is not known.

Objective:

To (1) test causally whether answer position shapes LLM validator consensus, (2) characterize pool-level validator biases, and (3) generate physician-rated estimates of clinical validity as hypothesis-generating pilot data.

Methods:

I generated 34,201 multiple-choice medication decision probes from MIMIC-IV discharge summaries. Three LLM validators (GPT-4o, Claude Sonnet 4.6, Gemini 2.5 Flash) answered each probe; 17,815 consensus-accepted probes (Cohen κ between provider pairs = 0.63) formed the accepted pool. Three analyses: (1) a pre-registered shuffling experiment randomly reassigned the correct-answer position on 2,000 probes while holding clinical content constant; validators were re-run on both variants in a paired design. (2) Pool-scale validator behavior on all 34,201 probes was analyzed for specialty, task-type, severity, and answer-position effects. (3) Two independent board-certified physicians audited 100 consensus-accepted probes; Cohen κ is the primary agreement metric, with per-physician, strict-consensus, and either-invalid rates reported with Wilson 95% CIs.

Results:

Position-shuffling (n=2,000): Consensus acceptance varied 24.6 percentage points across shuffled positions (A: 61.4%, B: 67.2%, C: 77.7%, D: 86.0%; χ²=89.82, P=2.4×10⁻¹⁹), providing causal evidence that answer position shapes validator consensus on ambiguous probes. Overall consensus acceptance dropped modestly from 77.0% (95% CI 75.1-78.7%) to 73.1% (71.1-75.0%) (McNemar P=3.1×10⁻⁵). Flip analysis showed 83-90% of answer changes followed content and only 3-7% followed position; the position gradient is therefore driven by a subset of near-indifferent probes rather than uniform position dependence. Pool-level biases (n=34,201): Validator acceptance varied by specialty (χ²=117.9), task type (χ²=168.2), severity (χ²=43.6), and observational answer position (χ²=196.8); all P<.001. Two-rater physician pilot (n=100): Cohen κ=0.25 between two board-certified physicians indicates substantial rubric-interpretation variability on medication decision MCQs; per-rater invalidity ranged 43-84%, too wide to support a point estimate. The pilot is hypothesis-generating and motivates larger multi-center replication.

Conclusions:

Answer position causally influences multi-provider LLM validator consensus on EHR-derived medication benchmarks, concentrated in near-indifferent probes but large enough to produce a 24.6 percentage point gradient; this extends prior individual-model position bias findings into the consensus layer. Pool-level biases across specialty, task type, and severity are strong. The two-rater pilot is hypothesis-generating, and points to rubric-interpretation variability requiring larger multi-center panels before population-level invalidity claims can be made. Benchmark designers using multi-LLM consensus should measure and correct for position and distributional biases. Clinical Trial: Not applicable. The pre-registration for the position-shuffling experiment is deposited on Zenodo (DOI: 10.5281/zenodo.19646578).