JMIR Publications

ABSTRACT

Background:

Most patient education materials in the United States are written at a reading level well above national recommendations, typically averaging between 9th-12th grade, while guidelines recommend a 6th grade level or lower. When patient education materials are written at a reading level that is too high, it can negatively impact patient care, leading to poorer health outcomes.

Objective:

The objective of this study was to evaluate the effect of explicit fifth-grade readability prompting on the linguistic complexity of large language model-generated patient education materials for common chronic conditions.

Methods:

In our study, we collected responses from ChatGPT, Copilot, and Gemini to 12 patient-centered questions regarding common chronic health-conditions, diabetes, cancer, and heart disease (3 chronic conditions x 4 questions each). Additionally, we structured the study into two arms: a prompted arm in which LLMs were instructed to generate responses at a 5th-grade reading level, and an unprompted arm in which no readability instruction was given (3 LLMs x 12 questions, prompted vs. unprompted). Readability scores were calculated using established indices and compared across LLMs and prompt types using linear mixed-effects regression with a random intercept for question.

Results:

Prompting LLMs to generate 5th grade responses produced substantially lower readability grade levels compared with unprompted outputs across all indices. For Flesch-Kincaid, prompted LSMeans ranged from 5.34 to 7.67, whereas unprompted LSMeans ranged from 11.47 to 11.98. Similar patterns were observed for SMOG (prompted 5.37-6.50 vs unprompted 10.60 - 10.88) and Gunning Fog (prompted 6.46 -7.97 vs unprompted 12.18 - 12.84). A significant LLM x Prompt Type interaction was observed for all readability indices (F = 3.34–4.88, p = 0.0417 to 0.0113) indicating that the magnitude of improvement differed across models.

Conclusions:

Prompting LLMs to use a 5th-grade reading level substantially improves the readability of their outputs, though the degree of improvement varies across models. These findings highlight the importance of prompt design and model selection when using LLMs for patient education.