JMIR Publications

ABSTRACT

Background:

Hard Flaccid Syndrome (HFS) is a rare and poorly understood andrological condition with limited epidemiologic data and no standardized diagnostic or therapeutic guidelines. As patients increasingly rely on artificial intelligence (AI) tools for health-related information, it is essential to evaluate the reliability of such tools in the context of rare disorders.

Objective:

This study aimed to assess the accuracy, readability, and reproducibility of ChatGPT responses to common patient-centered questions regarding HFS.

Methods:

Sixteen clinically relevant questions across four domains (epidemiology, symptoms and case presentation, diagnostics, and treatment) were developed from authentic patient queries. Questions were independently submitted to ChatGPT in separate chat sessions and repeated after 48 hours to assess short-term reproducibility. Responses were evaluated independently by three urologists with expertise in male sexual health, with discrepancies adjudicated by a fourth reviewer. Accuracy was graded using a four-point validated scale, inter-rater reliability was assessed using intraclass correlation coefficients, and readability was measured using the Gunning Fog Index (GFI).

Results:

Mean readability scores remained stable between evaluation sessions (GFI 15.19 ± 1.68 at T1 vs. 14.59 ± 1.66 at T2; p = 0.331), corresponding to an upper high school to early college reading level. No significant differences in readability were observed across domains. Inter-rater reliability for accuracy assessments was fair at the individual reviewer level and moderate when ratings were averaged. Overall reproducibility between sessions was high, with 14 of 16 responses (87.5%) receiving identical accuracy classifications.

Conclusions:

ChatGPT demonstrated stable readability and high short-term reproducibility when addressing HFS-related questions, but accuracy varied across clinical domains, reflecting both model limitations and gaps in the existing medical literature. While AI tools may provide accessible general information, their use in rare and poorly defined conditions should be approached cautiously and supplemented by expert clinical guidance.