Currently submitted to: JMIR Dermatology
Date Submitted: Aug 18, 2026
Open Peer Review Period: Aug 26, 2026 - Oct 21, 2026
(currently open for review)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Reliability, Transparency, and Readability of Large Language Model Chatbots as Auxiliary Information Tools for Hidradenitis Suppurativa: Cross-Sectional Study
ABSTRACT
Background:
Hidradenitis suppurativa (HS) is a chronic, debilitating, and frequently stigmatizing inflammatory skin disease characterized by a prolonged interval to diagnosis, which drives patients toward digital health resources and large language model (LLM)–based chatbots as auxiliary sources of medical information. However, the reliability and readability of chatbot-generated information on HS remain uncertain.
Objective:
This study aimed to systematically evaluate the reliability, transparency, and readability of four widely used AI chatbots as supplementary informational tools for patients with HS.
Methods:
The eight most popular HS-related search keywords worldwide, identified through Google Trends, were entered verbatim and individually into ChatGPT, Gemini, Perplexity, and Microsoft Copilot in independent, anonymized sessions (32 responses in total). Response reliability and structural quality were assessed using the DISCERN instrument, the Ensuring Quality Information for Patients (EQIP) tool, the JAMA benchmark criteria, and the Global Quality Scale (GQS). Readability was quantified using six validated metrics, including the Flesch Reading Ease Score (FRES) and the Automated Readability Index (ARI), benchmarked against the recommended sixth-grade reading level. Between-model differences were analysed using the Kruskal–Wallis test with Dunn–Bonferroni post hoc comparisons.
Results:
Reliability differed significantly across the four chatbots for DISCERN (P=.03, ε²=0.217), EQIP (P=.004, ε²=0.379), and JAMA (P<.001, ε²=0.659) scores, but not for GQS (P=.53). Perplexity achieved the highest median DISCERN score (45.00), and ChatGPT the highest median EQIP score (72.50); Microsoft Copilot scored lowest on both instruments. Post hoc analyses confirmed that Perplexity significantly outperformed Microsoft Copilot on DISCERN (P=.04) and EQIP (P=.02). In contrast, none of the chatbots met the recommended sixth-grade readability threshold: all grade-level indices substantially exceeded grade 6, and all FRES medians fell far below the target of ≥80 (range 20.34-44.96).
Conclusions:
Perplexity and ChatGPT generated comparatively reliable and well-structured HS information and show promise as auxiliary educational tools; however, all evaluated chatbots produced text far too complex for the average patient. Future model development should incorporate adaptive plain-language constraints so that clinical accuracy is matched by accessibility across all levels of health literacy. Clinical Trial: null
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.