JMIR Publications

ABSTRACT

Background:

High-quality observation and feedback contribute to the development of clinical competence and professional growth in medical education. Faculty often struggle to translate verbal observations into written feedback because of documentation burden and competing demands. Ambient artificial intelligence (AI) scribes, already adopted in clinical practice, may address this challenge by capturing verbal exchanges and generating structured notes.

Objective:

The purpose of this study was to examine the use of ambient AI scribes to generate educational feedback notes during a formative medical interviewing workshop for first-year medical students in spring 2025.

Methods:

Thirteen instructors were randomized to control (human-only) or intervention (AI scribe-assisted) workflows to complete narrative feedback forms. The intervention group used an AI scribe to generate transcripts of student-instructor encounters, which were then summarized into feedback notes using a large language model, and edited by instructors before submission. All narratives were scored using the Evaluation of Feedback Captured Tool (EFeCT). Factual accuracy of a subsample of unedited AI feedback summaries was reviewed against source transcripts. Task load and usability were measured using NASA Task Load Index and System Usability Scale respectively.

Results:

Instructors submitted feedback on 94 of 102 students (92.2%). Median EFeCT scores on the zero-to-five scale were higher for human-edited AI narratives (3.00) and unedited AI summaries (3.00) compared to human-only narratives (2.00; p<0.001). Human narratives were shorter than AI-assisted outputs (p<0.001). Review of 117 AI-generated feedback elements showed a 6.8% mischaracterization and 1.7% hallucination rate, with most errors corrected during editing. Task load was high and usability marginal in both control and intervention groups, with no significant differences.

Conclusions:

An ambient AI scribe-assisted workflow improved the quality of written narrative feedback with no observed increase in instructor effort compared to human-only documentation. Although occasional inaccuracies required review, this innovation has the potential to transform feedback documentation. Clinical Trial: This study received exemption from review by the Yale University Institutional Review Board due to its educational nature on January 23, 2025 (IRB Protocol ID 2000039478).