JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are increasingly used to obtain medical information, but the accuracy and readability of their responses to common questions about constipation remain uncertain.

Objective:

This study aimed to compare the accuracy and readability of five contemporary LLMs in providing information on constipation.

Methods:

A standardized bank of 37 guideline-derived constipation-related questions covered basic knowledge, diagnosis and evaluation, treatment, and prevention and special populations. Responses from GPT-4o, GPT-5.5, Gemini 3.1 Pro, Claude Sonnet 4.6, and DeepSeek V4 Pro were anonymized, randomized, and independently evaluated by three gastroenterologists using a 5-point Likert scale. Readability was assessed using the Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, and Simple Measure of Gobbledygook index.

Results:

Agreement among the three gastroenterologists was high across the five LLMs (Gwet's AC2 of 0.92-0.95). Overall accuracy was high (mean scores ranging from 4.42 to 4.74 across models) with GPT-4o scored slightly lower than GPT-5.5 (adjusted P=.005) and DeepSeek V4 Pro (adjusted P=.028). Domain-specific mean scores ranged from 4.04 to 4.85, but no adjusted pairwise comparison was significant. Mean Flesch-Kincaid Grade Level scores ranged from 12.36 to 14.58, and all four readability indices indicated that the responses generally required upper-secondary to college-level reading proficiency.

Conclusions:

All five LLMs provided general highly accurate answers to constipation-related questions, regarding information about basic knowledge, diagnosis and evaluation, treatment, and prevention and special populations. However, their responses required relative high levels for patient education. Further improvements are needed in optimization of readability, alignment with clinical guidelines, and clinician oversight.