JMIR Publications

ABSTRACT

Background:

Ambient artificial intelligence (AI) scribes for chart documentation have seen rapid adoption, but their educational impact on medical students has not been described.

Objective:

The purpose of this study is to determine the impact of an AI scribe on pre-clerkship medical student note writing.

Methods:

In this prospective non-randomized pre/post design study, all first-year medical students (n=104) at a single U.S. medical school submitted “human-only” notes based on a summative observed structured clinical examination (OSCE) station in May 2025. An AI scribe generated independent AI notes post-OSCE. A sub-group of students (n=47) consented to complete a second “hybrid” note by revisiting their human-only note and incorporating AI notes as perceived necessary, followed by a brief survey about the AI notes. Trained, blinded raters were randomly assigned to score all notes on 10 elements using QNOTE acceptability criteria (0=Unacceptable, 50=Partially, 100=Fully).

Results:

Across all elements, median evaluation scores of human-only notes were high (range 81.3 - 100) and were similar between students who participated in “hybrid” notes and those who did not. In paired analyses between “human-only” and “hybrid” notes, the only notable element-level change was a decline in Chief Complaint scores (P=0.05). Of the twelve participants in the bottom quartile of “human-only” QNOTE scores, 9 demonstrated improvement with “hybrid” scores. Participants agreed that the AI note “was more concise than my note” (78.7%) and would be “helpful as a first draft” (66.0%). About half (55.3%) agreed that the AI note “left out important details.” One in five (21.3%) agreed that the AI note “may reduce my ability to learn how to write a good note.”

Conclusions:

Interaction with AI notes among pre-clerkship medical students had little impact on quality of “hybrid” notes. Chief Complaint scores likely declined due to conciseness in AI notes that often omitted symptom duration. Our findings suggest that among students who predominantly write close to fully acceptable “human-only” notes, there was no detriment to clinical reasoning, and students were discerning in balancing AI’s conciseness and its omissions. Especially for lower-performing students, AI scribes could enhance students’ own note writing, though educational safeguards are necessary given the potential for harm due to overreliance on automated systems.