JMIR Publications

ABSTRACT

Background:

Large Language Models (LLMs) demonstrate strong performance at medical specialty board multiple choice question (MCQ) answering but underperform in more complex medical reasoning scenarios. This gap indicates a need to improve both LLM medical reasoning and evaluation paradigms.

Objective:

To develop and evaluate MedEvalArena, a dynamically regenerable benchmark assessing LLM medical reasoning.

Methods:

In a symmetric round-robin format, six frontier LLMs from OpenAI, Grok, Gemini, Claude, Kimi, and DeepSeek families generated challenging board-style medical MCQs, served in an ensemble LLM-as-judge panel to adjudicate validity of generated questions, and answered the validated exam as examinees. Candidate items were generated until each model contributed 50 validated questions according to AAMC/ABMS physician specialty distributions, yielding a 300-question exam. The judge panel adjudicated candidate question logical validity and medical accuracy. We compared LLM question-generation validity rates, assessed leave-one-out judge agreement, and tested for self-judging preference. Questions were audited for overlap with public medical QA benchmarks. We tested question-answering accuracy across LLMs using binomial-logit generalized linear mixed model (GLMM) likelihood ratio test (LRT). A generator-stratified 90-question subset was completed by five board-certified physicians for paired human-LLM analysis. A sensitivity analysis assessed the effect of varying the LLM-as-judge medical accuracy score threshold. We also evaluated inference cost–accuracy tradeoffs. Multiple comparisons were adjusted by the Benjamini-Hochberg procedure (BH).

Results:

Questions generated by frontier models from OpenAI (94.8%), Gemini (92.3%), and Claude (83.3%), respectively, each had higher judge panel validity rates compared to Kimi (63.8%), Grok (56.0%), and DeepSeek (46.0%) models (two-sample proportion test, P<.01). The public-source similarity audit found no exact or near-duplicate matches. Leave-one-out judge agreement was moderate. We assessed for self-judging bias and found that generator-specific same-family judging coefficients were non-negative (0 to 11.0 percentage points [pp]), however, none remained significant after BH adjustment. There was no significant difference between the six frontier LLM question-answering accuracies, which ranged from 85.7% to 91.7%, on the 300-question exam (GLMM LRT P=.065). However, the study was powered at 80% to detect LLM vs LLM accuracy difference of only 9.2 pp, so this result should not be interpreted as equivalence. Across the 90-question subset, physician mean accuracy was 85.6% (CI 79.4%-91.7%), compared with aggregate LLM performance of 94.4% (85/90, CI 87.5%-98.2%). This difference was not significant (McNemar’s test, P =.077), however, the study was powered to detect human vs LLM accuracy difference of 14-15%. When jointly considering accuracy and inference cost, answering-only costs varied two orders of magnitude and multiple models lie on the Pareto frontier, with no single model dominating across both dimensions.

Conclusions:

MedEvalArena enables scalable, regenerable, peer-judged benchmarking of LLM medical reasoning. In this study, question generation validity differentiated model families more clearly than answering accuracy, suggesting that valid MCQ generation may complement QA benchmarks for assessing LLM medical reasoning.