This paper has been accepted and is currently in production.
It will appear shortly on 10.2196/99780
The final accepted version (not copyedited yet) is in this tab.
Quality and Readability of ChatGPT and Gemini Responses to Patient Questions About Systemic Isotretinoin: Comparative Evaluation Study
ABSTRACT
Background:
The use of artificial intelligence(AI) in healthcare is rapidly expanding due to its accessibility, with growing reliance among both physicians and patients.
Objective:
This study evaluated ChatGPT(OpenAI) and Gemini (Google) responses to patient questions on systemic isotretinoin.
Methods:
Forty commonly anticipated patient questions were developed based on online sources and patient education materials and categorized into five groups. Responses were generated using the January 2025 free versions of ChatGPT-4o and Gemini 2.0 Flash. Five experienced dermatologists evaluated the responses in a blinded manner using a five-point Likert scale assessing accuracy, clarity, completeness, usefulness, and overall quality score (GQS). Readability was measured using Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease (FRE), and SMOG (Simple Measure of Gobbledygook) Index scores. Mixed-effects modeling was performed to adjust for reviewer and question variability, and ICCs (intraclass correlation coefficient) were calculated to assess inter-rater reliability.
Results:
ChatGPT achieved significantly higher ratings in accuracy, completeness, usefulness, and GQS, while clarity showed a non-significant trend. When questions were analyzed by subtype no statistically significant differences were observed between models across evaluation parameters. Mixed-effects modeling confirmed higher odds of superior ratings for ChatGPT across domains (ORs 1.82–5.88; all p<0.001). However, inter-rater reliability was lower for ChatGPT (global ICC 0.401) compared to Gemini (0.677). Readability analyses demonstrated that ChatGPT required a higher reading level (FKGL 12.6 vs. 11.2; SMOG 11.3 vs. 10.0).
Conclusions:
ChatGPT outperformed in terms of data accuracy and quality, while Gemini offered superior readability and consistency. Both models exceeded recommended readability thresholds, highlighting a gap between informational quality and linguistic accessibility in AI-generated health communication.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.