Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
A Multidimensional Comparative Study of Multiple Mainstream Large Language Models in Perioperative Consultation for Hypospadias Surgery
ABSTRACT
Background:
Hypospadias is a common congenital malformation requiring surgical correction, with caregivers facing significant perioperative information needs. Large language models (LLMs) offer a potential solution for health education, yet their performance in pediatric urological consultations remains unexplored.
Objective:
This study aimed to evaluate the performance of five LLMs—ChatGPT-4o, Gemini-2.5-Pro, OpenEvidence, Zhipu Qingyan, and DeepSeek—in addressing core concerns of caregivers during the perioperative period for children with hypospadias, and to assess their application value and limitations in pediatric urological clinical health education.
Methods:
A prospective, non-interventional, cross-sectional design was employed. A question bank was developed based on literature and clinical practice, with 10 high-priority questions selected via questionnaire screening for the test set. Seven experts (6 dimensions) and 32 caregivers (4 dimensions) were prospectively recruited to evaluate responses using a double-blind forced-order ranking method (reverse scoring from 1 to 5), with reference authenticity being verified.
Results:
Significant differences existed across dimensions among the five models (P<.001). Gemini-2.5-Pro demonstrated overall superior performance, ranking first in both expert evaluations (median 5.0 [IQR 4.0-5.0]) and caregiver evaluations (median 4.0 [IQR 3.0-5.0]), with outstanding structural capabilities; DeepSeek ranked second (median 4.0), demonstrating relatively consistent ratings across socioeconomic strata and superior emotional support compared to ChatGPT-4o (P=.001); OpenEvidence scored lowest (nearly 50% “poor” ratings), exhibiting poor readability but reliable evidence sources.
Conclusions:
Gemini-2.5-Pro offers the most comprehensive quality for perioperative hypospadias consultations. DeepSeek and Zhipu Qingyan demonstrate strong rapport in home care guidance but require strict control of literature hallucination risks. Perioperative care for hypospadias should adopt a tiered human-machine collaboration model based on clinical risk stratification, balancing communication efficiency and health education safety.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.