Currently accepted at: JMIR Medical Education
Date Submitted: Mar 5, 2026
Open Peer Review Period: Mar 5, 2026 - Apr 30, 2026
Date Accepted: Nov 6, 2026
(closed for review but you can still tweet)
This paper has been accepted and is currently in production.
It will appear shortly on 10.2196/94741
The final accepted version (not copyedited yet) is in this tab.
Rating the Quality of Intraoperative Surgical Feedback With a Multi-Instance Large Language Model Framework: Development and Validation Study
ABSTRACT
Background:
Verbal feedback delivered by attending surgeons in the operating room plays a critical formative role in resident trainee skill acquisition. Yet, assessing the quality of trainer feedback and its effectiveness in influencing trainee behavior during live surgery remains a challenge. Prior studies relied on extensive manual annotation by expert human raters and focused on broad taxonomies that overlook the qualitative aspects of feedback delivery such as clarity or urgency. Limited existing automated methods, including keyword analysis and topic modeling, also fail to capture these nuanced delivery dimensions.
Objective:
The study aimed to develop and evaluate a scalable, automated framework for discovering and scoring interpretable surgical feedback quality criteria grounded in real-world surgical training interactions and clinically validated outcome measures.
Methods:
We introduce a two-stage large language model (LLM)-based framework. In the first stage, multi-instance prompting, meaning repeated independent sampling with multiple GPT-4o instances seeded with clinically validated definitions of feedback effectiveness outcomes and unlabeled feedback examples, proposes candidate quality criteria. These are consolidated via hierarchical clustering and a deterministic LLM synthesis step into six human-interpretable, behaviorally anchored dimensions: Encouragement, Urgency, Actionability, Timeliness, Clarity, and Reflection. In the second stage, these criteria are applied to score feedback instances at scale using an LLM-as-a-judge approach. Framework evaluation included predictive modeling of four clinically annotated behavioral outcomes, statistical significance testing using DeLong's method, generalized linear mixed modeling of associations between quality dimensions and outcomes, and human-AI alignment assessment using quadratically weighted Cohen's kappa on a stratified sample of instances rated by two domain-expert human raters.
Results:
Applied to 4,210 intraoperative feedback instances, the six AI-discovered quality criteria achieved AUROCs of 0.75 (95% CI 0.74-0.77) for trainee behavioral adjustment and 0.71 (95% CI 0.69-0.72) for trainee verbal acknowledgment. Augmenting prior automated topic modeling features with our criteria yielded consistent gains of 9-12% for the two trainee outcomes and 3-11% for the two trainer outcomes. DeLong's testing showed that the AI-derived dimensions added predictive value for both trainee outcomes against both the topic modeling and human-annotated baselines, and for each trainer outcome against one of the two baselines. Generalized linear mixed modeling revealed that Actionability (rate ratio [RR]=1.22), Timeliness (RR=1.24), and Urgency (RR=1.11) were significantly associated with trainee behavioral adjustment, while Reflection (RR=1.34) and Clarity (RR=1.13) predicted verbal acknowledgment. Human-AI alignment was substantial for five of six dimensions (quadratically weighted kappa=0.60-0.79), approaching inter-human agreement levels.
Conclusions:
Our LLM-based framework enables scalable, interpretable, and clinically grounded assessment of surgical feedback delivery quality, allowing automated quality scoring after initial human calibration and validation without requiring manual quality annotation of every feedback instance. The discovered criteria demonstrate predictive validity for real-world trainee and trainer behavioral outcomes and exhibit strong alignment with expert human judgment, providing a foundation for improving intraoperative teaching and surgical education quality assurance.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.