JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) can propose scores for clinical short answers, but how model advice affects teachers' final ratings remains uncertain.

Objective:

To compare 3 complete scoring workflows and evaluate the effect of adding fixed LLM advice and interface training to a shared structured review protocol in new learners and teachers.

Methods:

Stage 1 analyzed 280 test responses from 70 gastroenterology trainees at 2 centers, each graded by 3 different teachers under independent human (H), conventional artificial intelligence (AI)-assisted (B), and structured AI-assisted (T) workflows (840 ratings across 12 teachers). Stage 2 enrolled 20 new learners (60 responses) and 12 new teachers in 6 within-center matched pairs, assigning one teacher within each pair by lot to structured review alone (S) and the other to the same review plus fixed archived advice and interface training (SAI). Both stages evaluated final-score mean absolute error (MAE) against an expert reference (100-point scale) using material and teacher fixed effects with learner-clustered CR2/Satterthwaite inference and Holm adjustment (3 contrasts) in stage 1, and exact paired randomization inference (64 allocations) in stage 2.

Results:

Stage 1 final MAE was 15.11 for H, 17.34 for B, and 5.13 for T. Adjusted differences were −9.92 for T−H (95% CI −11.98 to −7.86; Holm P<.001), −12.14 for T−B (95% CI −13.96 to −10.32; Holm P<.001), and 2.22 for B−H (95% CI −0.003 to 4.44; Holm P=.0503); against initial experts, T−B was −8.19 to −7.62. Stage 2 completed all 240 ratings (advice viewed in 116/120 SAI tasks). Final MAE was 12.92 for S and 7.00 for SAI; all 6 pairs favored SAI (equal-pair difference −5.92, nominal 95% randomization-based CI −12.00 to −1.75; exact P=.03125). Workflow time differed by −5.38 seconds (95% CI −38.20 to 16.60; P=.59). On the 59 responses with advice (post hoc), MAE was 3.56 for LLM alone, 7.03 for SAI, and 12.97 for S; item counts showed both correct and incorrect advice in final scores.

Conclusions:

When the structured review protocol was held constant, adding fixed LLM advice and interface training after a locked initial score reduced teachers' final scoring error across 6 matched pairs, without evidence of time savings. Displaying LLM scores alone did not improve grading accuracy; benefit appeared when advice was embedded in a review protocol requiring an independent initial judgment. Clinical Trial: Not publicly registered.