JMIR Publications

ABSTRACT

Background:

Background:

Large language models (LLMs) may support clinical documentation; however, comparative evidence on Japanese subjective, objective, assessment, and plan (SOAP) note generation from transcripts of actual outpatient encounters remains limited.

Objective:

Objective:

We aimed to compare three LLMs, across three generation runs, for Japanese SOAP note generation from automatically transcribed outpatient encounters and to assess the agreement between physician ratings and a language-model judge.

Methods:

Methods:

This single-center retrospective evaluation used 154 deidentified automatic speech recognition (ASR) transcripts from three outpatient departments in Japan. Qwen2.5-14B-Instruct, Gemini 2.5 Pro, and Claude Opus 4.6 each generated three notes per transcript in three independent runs under common instructions (1386 notes), all of which were evaluated. Two physicians created reference notes from the same ASR text; the original audio was unavailable. Three other physicians, blinded to model identity, rated all notes on nine attributes of an adapted Provider Documentation Summarization Quality Instrument and screened them for four categories of severe documentation error. Models were compared using mixed-effects models. GPT-4o, used as an LLM-as-a-judge, scored eight attributes with generic and, in a second run, the physicians' anchors; agreement with mean physician ratings was assessed using absolute-agreement intraclass correlation coefficients (ICCs) and Bland-Altman analysis. Analyses were exploratory and unadjusted for multiplicity.

Results:

Results:

Mean nine-item scores were 4.02 (SD 0.38) for Qwen, 4.26 (SD 0.33) for Gemini, and 4.38 (SD 0.32) for Claude (omnibus P<.001); the reliability of the three-physician mean was ICC(2,3)=0.84 (95% CI 0.81-0.87). At least one severe error was identified in 85/462 (18.4%; 95% CI 15.1%-22.2%), 57/462 (12.3%; 95% CI 9.6%-15.7%), and 35/462 (7.6%; 95% CI 5.5%-10.4%) notes, respectively (omnibus P<.001); the 95% CIs of all three model-based pairwise odds ratios excluded 1, and critical omission was the most frequent category. The ordering of the models was the same in each run, and 33.8%, 23.4%, and 14.9% of transcripts had a severe error in at least one run. Severe errors were present in 18.6%, 11.1%, and 7.4% of notes rated clinically useful. The judge-physician ICC for the eight-item score was 0.68 (95% CI 0.61-0.74) with generic anchors and 0.71 (95% CI 0.65-0.76) with the physicians' anchors. Of the 177 notes with a severe error, 85 (48.0%) were attributed to features of speech-recognized dialogue, such as speaker-role misattribution, misresolution of omitted subjects in Japanese, and suspected speech recognition errors.

Conclusions:

Conclusions:

The models differed in physician-rated quality and in the proportion of notes with severe documentation errors, and severe errors occurred with every model in every run, including in notes rated useful. These findings support prospective evaluation of physician-reviewed drafting but do not establish the safety of complete audio-to-note pipelines or workflow benefits. Language-model scoring agreed moderately with physician ratings and may complement, but not replace, physician assessment.