JMIR Publications

ABSTRACT

Background:

Background:

Patients with infertility often have inadequate reproductive health literacy (RHL), while existing health education materials may be difficult to read, overly complex, and insufficiently tailored to individual needs. In recent years, large language models (LLMs) have created new opportunities to improve patient health education through their ability to generate and interpret medical information. However, their applicability and content quality in infertility education remain insufficiently evaluated.

Objective:

Objective:

This study systematically evaluated the readability and content quality of LLM-generated health education materials for patients with infertility, identified model strengths and limitations, and provided evidence for their standardized use in reproductive health.

Methods:

Methods:

This study selected DeepSeek-R1, ChatGPT 5.2 Plus, Kimi 2.5, Qwen 3, and Gemini 3 as the study subjects. Based on the theoretical framework of RHL, this study developed a standardized question list comprising 6 dimensions and 93 items, and used unified prompts to generate health education texts in both Chinese and English. In this study, the Datayze Readability Analyzer was used to assess text readability via the Flesch Reading Ease (FRE), Gunning Fog Index (GFI), and Flesch-Kincaid Grade Level (FKGL). Meanwhile, the content quality was evaluated in a double-blinded manner by reproductive medicine experts from three dimensions: accuracy, completeness, and humanistic care. Repeated-measures analysis of variance was used to compare performance across the LLMs, and Pearson correlation analysis was conducted to examine the associations between readability and content quality. A composite radar chart was used to evaluate the overall performance of each model.

Results:

Results:

The five LLMs differed significantly in both text readability and content quality (P<0.001). In terms of readability, Gemini 3 and Qwen 3 performed best, with FKGL scores below the target threshold corresponding to a sixth-grade reading level. DeepSeek-R1 and Kimi 2.5 performed least favorably, with FKGL scores corresponding to a ninth-grade reading level. ChatGPT 5.2 Plus showed intermediate performance. In terms of content quality, Kimi 2.5 performed best across accuracy, comprehensiveness, and humanistic care, followed by DeepSeek-R1. ChatGPT 5.2 Plus performed least favorably. Further analysis revealed significant negative correlations between readability and content comprehensiveness for texts generated by Kimi 2.5 and Gemini 3 (P<0.05), indicating a trade-off between these two attributes. The composite radar chart analysis showed that Kimi 2.5 achieved the highest overall score (11,594.33), whereas ChatGPT 5.2 Plus had the lowest score (5,555.83).

Conclusions:

Conclusions:

Different LLMs showed significant differences in performance in generating health education texts for patients with infertility. In some models, readability was negatively correlated with content comprehensiveness. Under the conditions of this study, Kimi 2.5 demonstrated strong overall performance. However, no model achieved the best performance across all evaluation dimensions. Future applications of LLMs in reproductive health education will require domain-specific optimization, context-specific adaptation, and systematic quality control to support more accurate, comprehensive, and patient-centered communication.