JMIR Publications

ABSTRACT

Background:

The Korean National Health Screening Program assigns each laboratory and anthropometric result to a tier of Normal A, Normal B (borderline), or suspected disease. Large language models (LLMs) are increasingly used to interpret laboratory results, but their ability to perform national-program tier determination accurately and reproducibly, particularly near decision boundaries, has not been evaluated.

Objective:

This study aimed to construct and release a dual-verified benchmark for Korean health-checkup determination and to evaluate rule grounding, zero-shot failure patterns, and test–retest reliability in 2 frontier LLMs.

Methods:

Using published criteria, we generated a tier-classification set of 144 cases across 20 items and a value-ladder set of 528 cases in 22 ladders, yielding 6,321 ordered pairs. Gold labels were independently verified by a deterministic rule engine (100% agreement; 0/6,321 violations). Claude Fable 5 and GPT-5.6 Sol were evaluated under zero-shot and rule-grounded conditions in 10 runs per model per condition. Each run was a stateless full-benchmark pass; run-level inference concerned stochastic evaluations of fixed cases, not new clinical samples. The primary outcome was monotonicity violation rate; key secondary outcomes were the rule-grounding contrast and test–retest reliability. Analyses used ladder-cluster bootstrap, exact unpaired rank tests across runs, and paired case-level McNemar tests. BMI stratification was pilot-informed and specified before the main 10-run evaluation.

Results:

Under rule grounding, every observed run achieved 100% tier accuracy and 0 monotonicity violations, whereas no zero-shot run achieved 100% accuracy (exact unpaired run-level rank test, P<.001 per model). The paired case-level majority-vote comparison was not statistically significant (McNemar P=.0625), the minimum attainable 2-sided value with 5 one-directional discordant pairs. Zero-shot failures showed opposite model/configuration-specific patterns: excluding body mass index, GPT produced 4.5 times as many violations as Claude (P=.01), whereas on body mass index ladders, Claude produced 2.5 times as many as GPT (P<.001); the aggregate comparison showed no difference (P=.99). At equal run counts, flip proportions were 6.4% for Claude and 11.2% for GPT (McNemar P=.004). Ten runs met a development-stage, study-specific SEM criterion for all nonzero zero-shot ladder estimates examined; this was not a prospectively prespecified or universal minimum.

Conclusions:

For the public rules evaluated, no errors were observed when the criteria were supplied in context, whereas zero-shot errors were model/item specific and aggregate accuracy obscured reliability differences. These findings support explicit rule grounding and repeated runs with justified sufficiency for evaluating this prerequisite rule-application capability; they do not establish end-to-end clinical deployment safety.