JMIR Publications

ABSTRACT

Background:

Large language models are increasingly used to generate health education materials, yet questions remain about whether LLM-generated content can balance professional accuracy with public accessibility, and what ethical challenges may arise during deployment.

Objective:

To evaluate Chinese pediatric asthma educational materials generated using large language models, comparing AI-generated materials (AI-GRs) against published expert-authored responses (PEARs) on total scores, adoption intent among healthcare professionals, perceived usefulness among family members, and the effect of AI-simplified versions (AI-SRs), as well as to assess participants’ ability to correctly identify the source of materials.

Methods:

In this randomized, double-blind evaluation study, participants (medical professionals and patient family members) were randomly assigned (1:1:1) to evaluate published expert-authored responses (PEARs), AI-generated responses (AI-GRs), or AI-simplified responses (AI-SRs). Each participant assessed five randomly selected items from their assigned set using a 19-item, 5-point Likert scale based on the Information Adoption Model. Secondary outcomes included dimension-specific scores, source identification accuracy, and readability indices.

Results:

AI-GRs had numerically higher mean total scores on the 19-item questionnaire than PEARs among both medical professionals (76.17 vs 72.97; Holm-adjusted P=.532) and pediatric patient families (68.77 vs 64.18; Holm-adjusted P=.149). The corresponding mean scores for AI-SRs were 72.98 and 65.47, respectively, with no statistically significant differences from PEARs after Holm adjustment (both adjusted P=1.000). No statistically significant differences were detected between either LLM group and PEARs in any of the four questionnaire dimensions after Holm adjustment. The material group–population interaction was also not statistically significant (F2, 513=0.116, P=.891, partial η²<.001). Regarding source identification, 80.6% of participants assigned to PEARs identified the material as human-authored, whereas only 7.3% of those assigned to the LLM groups identified the material as LLM-generated.

Conclusions:

This randomized, double-blind evaluation study provides preliminary participant-level evidence regarding the perceived quality and acceptance of LLM-assisted pediatric asthma education among medical professionals and pediatric patient families under controlled conditions. Additional language simplification did not improve acceptance in this predominantly highly educated sample, and the strong tendency to attribute materials to human authors highlights the importance of source transparency. Professional review and dedicated assessments of factual accuracy, clinical safety, comprehension, and behavioral outcomes remain necessary before practical implementation. Clinical Trial: Chinese Clinical Trial Registry ChiCTR2500112000; https://www.chictr.org.cn/showproj.html?proj=255052. The study was retrospectively registered on November 9, 2025, after participant recruitment had begun.