JMIR Publications

ABSTRACT

Background:

Clinicians spend a lot of time on Electronic Health Record (EHR) documentation, often at the expense of direct patient interaction. Ambient listening technology uses artificial intelligence to passively record and summarize clinical encounters. While early studies are promising, there is limited evidence on system performance in complex, non-English settings.

Objective:

To compare the documentation performance of three commercially available ambient listening systems in simulated Dutch-language outpatient consultations by assessing note completeness, correctness, and conciseness under predefined linguistic and interactional challenges.

Methods:

Standardized audio recordings of 10 scripted physician–patient interactions across 2 specialties were used. Scenarios included multi-speaker dynamics (patient companion), conversational disruptions (nurse interruption), evasive patient communication, and a regional dialect (Gronings). Three ambient listening providers (Provider A, Provider B, and Provider C) processed the audio files. Generated notes were divided into 4 note sections: patient history, physical examination, diagnostic impression, and treatment plan. For each system output, each note section was scored for completeness, conciseness, and correctness using a 5-point ordinal scale against physician-developed reference summaries. In total, 8 clinicians participated in scoring: 4 junior clinicians evaluated all outputs, and 4 senior clinicians each reviewed a subset of scenarios. Analyses were based on 351 rater-averaged section-domain scores. Inter-rater agreement was assessed using Gwet AC2. System-level technical characteristics were assessed alongside clinical performance to aid interpretation of between-provider differences.

Results:

Across 351 rater-averaged section-domain scores on a 1-5 scale, overall inter-rater agreement was high (Gwet AC2 = 0.827). Mean scores were tightly clustered across providers (Provider C: 4.26, Provider B: 4.00, Provider A: 3.82). Mean scores were higher in Otolaryngology (mean 4.36) than Surgical Oncology (mean 3.68). Across scoring domains, correctness received the highest mean score (4.21), while completeness received the lowest (3.81). Variation in mean scores was observed across script scenarios. Dialect-containing scenarios had the lowest mean score overall (3.77), although this pattern was not uniform across specialties and systems. Median summary generation times ranged from 13.5 seconds (Provider C) to 22.0 seconds (Provider B).

Conclusions:

Ambient listening systems demonstrated generally strong performance in this simulated Dutch-language evaluation, including under several common conversational challenges. However, mean scores varied across scripted scenarios, with lower scores observed in some dialect-containing and otherwise challenging interactions. Future deployment studies must prioritize linguistic equity, real-world validation of efficiency gains, and evaluation of both clinician and patient perception to understand how these systems influence consultation dynamics and care delivery across diverse patient populations.