JMIR Publications

ABSTRACT

Background:

Large language model (LLM) benchmarking in clinical medicine is increasingly reliant on LLM-based assessment frameworks; in these systems, one artificial intelligence scores the outputs of another.

Objective:

This practice relies on the assumptions of evaluator neutrality and rubric consistency. These assumptions have not been empirically tested in clinical fields. In this study, we aim to directly examine these assumptions by using the PARM (Planning-Action-Reflection-Memory) framework as a theoretical lens to characterize endovascular device selection and clinical reflective reasoning as a clinical field.

Methods:

We developed a scoring rubric (C1-C6) with 40 clinical vignettes and six criteria across five endovascular fields (peripheral artery disease, abdominal aortic aneurysm, chronic venous insufficiency, coronary in-stent restenosis (ISR), and cerebrovascular/neuroendovascular; 8 vignettes per field). Four LLMs were evaluated as clinical decision support systems: GPT-4o (M1), Claude Sonnet 4.6 (M2), Gemini 1.5 Pro (M3), and Llama 3.3 70B (M4). Two LLMs served as independent raters with the same evaluation criteria: Claude Sonnet 4.6 (Rater 1) and GPT-4o (Rater 2). The rater identity was intentionally disclosed; the primary research question concerned rater scoring behavior rather than the absolute performance of the model. Human assessment was intentionally excluded. Human evaluation was intentionally excluded because its inclusion would have shifted the study question from rater disagreement to rater accuracy, which represents a distinct secondary question. Inter-rater reliability was measured using ICC(2,1) two-way random effects modeling.

Results:

The combined ICC(2,1) was 0.168 (95% CI: 0.017–0.312). Consistent with the criterion-interpretation divergence hypothesis, this low ICC was not random: Raters produced very close scores for M1 and M2 (Δ = −9 and +5, respectively), but differed significantly for M3 Gemini (Δ = −157 points). Scoring behavior analysis identified two intrinsically consistent but mutually inconsistent criterion-interpretation models. Rater 1's scores were most consistent with the reasoning quality approach (partial credit to clinical competence independent of vignette context). Rater 2's scores were most consistent with the context fidelity approach (near-zero scores for wrong vignette responses independent of reasoning quality). Contrary to the self-preference hypothesis, both raters placed the other higher-performing model above their own. Both raters independently identified C6 (PARM Reflection) as the weakest criterion across all models and V-NEURO-05 as the most significant distinguishing vignette. This pattern was consistent across both raters despite their differing scoring approaches, suggesting it may reflect a systematic tendency rather than an artefact of a single evaluator's criteria.

Conclusions:

AI raters used in LLM-based clinical benchmarking do not behave as interchangeable scoring tools. Their scoring behavior is most consistent with model-specific criterion interpretation patterns that systematically produce different outcomes. The observed low inter-rater reliability requires interpretation rather than simply being ignored: it reflects structured, direction-specific disagreement concentrated in models with context-tracking failures, not random inconsistency. The independent convergence of both raters on the PARM Reflection deficit is the most robust finding of this exploratory study. All four evaluated models demonstrated high guideline knowledge but exhibited limited spontaneous reflective reasoning in clinical decision scenarios.