JMIR Publications

ABSTRACT

Background:

Short answer questions (SAQs) effectively assess higher-order reasoning skills in medical education but face challenges in scalability due to time-intensive manual grading, and inter-rater reliability. Recent advances in large language models (LLMs) may offer a solution through automated assessment.

Objective:

This study evaluates the performance and consistency of LLMs—ChatGPT, Claude–Sonnet4, and Mistral pro—compared to human graders for SAQs grading in hepatology and gastroenterology.

Methods:

We retrospectively analyzed SAQs from hepatology and gastroenterology examinations at Sorbonne University (2023–2025). Three human evaluators (two hepato-gastroenterologists and one radiologist) independently graded all responses twice, with majority consensus score serving as the reference standard. Six LLMs were evaluated: ChatGPT (GPT-4o, GPT-4.5, GPT-4o3, GPT-5-Thinking), Claude–Sonnet4, and Mistral pro. Each LLM graded all responses twice using standardized prompts including questions, model answers, grading guidelines, and scoring rubrics. Agreement was assessed using Quadratic Weighted Kappa (QWK), Cohen's kappa, Intraclass Correlation Coefficient (ICC), and Mean Absolute Error (MAE) for inter/intra-human, LLM-versus-human, and intra-model comparisons.

Results:

902 student responses from 12 SAQs were included and analysed. Inter-human grader agreement was almost perfect (QWK = 0.90–0.92, ICC = 0.90–0.92, MAE = 0.05–0.07). Claude–Sonnet4 and GPT-4o3 demonstrated near-human-level performance when compared to the majority vote of human evaluators (Claude–Sonnet4: QWK = 0.88–0.90, ICC = 0.88–0.90, MAE = 0.06–0.07; GPT-4o3: QWK = 0.82–0.89, ICC = 0.82–0.89, MAE = 0.07–0.09). GPT-5-Thinking showed substantial agreement (QWK = 0.77–0.79). Mistral pro and older ChatGPT versions performed poorly (QWK < 0.63). Intra-model consistency was highest for Claude–Sonnet4 (QWK = 0.93), approaching the best human intra-rater reliability. Agreement did not correlate with prompt-design choices (number of examples or minimum word count).

Conclusions:

Advanced LLMs, particularly Claude–Sonnet4 and GPT-4o3, substantially reduce grading time while maintaining near-human agreement, potentially enabling more frequent use of SAQs in student evaluation. However, performance varies by model, and agreement alone does not ensure educational validity. Until broader multicenter validation across diverse domains, stakes, and question format–along with safeguards for bias, privacy, and model drift–LLMS should augment rather than replace human faculty graders.