Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Nov 16, 2025
Date Accepted: May 27, 2026
Readability of Artificial Intelligence-Generated Patient Visit Summaries in Orthopaedic Surgery: A Retrospective Analysis
ABSTRACT
Background:
Patient visit summaries (PVS) are patient-facing documents intended to reinforce communication and promote patient education after clinical encounters. However, current visit summaries are time-consuming to generate, lack personalization, and often exceed patient literacy levels. Artificial intelligence (AI)-based scribes can generate personalized PVS in real time, offering a potential solution.
Objective:
This study aimed to evaluate the readability of AI-generated PVS in an orthopaedic setting and determine their alignment with established literacy standards for patient-facing materials.
Methods:
We retrospectively analyzed 1,007 consecutive AI-generated PVS from an academic orthopaedic surgery outpatient clinic between December 2023 and May 2024. Readability was assessed using the Flesch-Kincaid Grade Level (FKGL), which estimates the U.S. grade level required to comprehend a text (lower scores = better readability), and the Flesch Reading Ease Score (FRES), which ranges from 0 to 100 (higher values = better readability). The proportions of PVS meeting the sixth- and eighth-grade benchmarks were also determined. The association between word count and readability was assessed using Spearman’s correlation (ρ). Kendall’s Coefficient of Concordance (W) was used to evaluate consistency and agreement between the different readability scores. Statistical significance was set at P<.05.
Results:
The AI-generated PVS had a mean FKGL of 9.3 (SD 1.3, 95% CI 9.3-9.4) and mean FRES of 57.3 (SD 8.3, 95% CI 56.8-57.8), corresponding to “fairly difficult.” Less than 1% (6/1,007) of PVS met the sixth-grade benchmark and 15.3% (154/1,007) met the eighth-grade threshold. Word count demonstrated weak positive correlation with FKGL (ρ=0.068, P=.030). Readability indices showed strong agreement (W = 0.90, P=.001).
Conclusions:
In this orthopaedic clinic, AI-generated PVS demonstrated ninth-grade readability on average, with most exceeding the sixth- and eighth-grade benchmarks. While AI-generated PVS may modestly improve upon traditional methods, further refinement and real-world comprehension testing are needed to optimize health communication. Clinical Trial: N/A
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.