JMIR Publications

ABSTRACT

Background:

Pediatric care resources are limited and unevenly distributed in many health systems. Caregivers increasingly turn to large language models (LLMs) for children’s health information, but real-world evidence on LLM use and response quality in general pediatric care is limited.

Objective:

To characterize real-world use, consultation needs, and response quality of Xiaoyi, an LLM-based pediatric health assistant, and to evaluate the impact of expert-guided optimization on response quality.

Methods:

This study collected all consultation sessions submitted to Xiaoyi between April 7 and December 31, 2025. Xiaoyi is a pediatric health assistant codeveloped by Children's Hospital of Chongqing Medical University and Zuoyi Tech. Temporal patterns of consultation activity were characterized; all consultation sessions were annotated for pediatric subspecialty, reason for consultation, and presenting symptoms through LLM-assisted labeling with pediatrician validation. To support iterative improvement, a multidisciplinary panel of 40 pediatricians evaluated Xiaoyi's response quality over two rounds: an initial evaluation informed system optimization (prompt revision, knowledge-base and tool expansion, and validation rules), and a follow-up evaluation assessed its impact. In each round, 1,000 randomly sampled sessions were rated at the session level (overall quality and clinical appropriateness) and the turn level (six predefined deficiency categories). Categorical ratings were compared using the chi-square test, and turn-level deficiency rates using generalized estimating equations to account for within-session correlation.

Results:

Caregivers of 8,266 children (median age 4.2, IQR 1.3–8.0 years; 4,705/8,266, 56.9% male) from 31 of 34 provincial-level divisions in China initiated 16,074 sessions, of which 43.8% (7,046/16,074) occurred outside routine clinic hours (08:00–17:30). Queries spanned 31 subspecialties, most frequently child healthcare (2,079/16,074, 12.9%), pulmonology (1,899/16,074, 11.8%), and gastroenterology (1,830/16,074, 11.4%). The leading reasons for consultation were symptom consultation (7,845/16,074, 48.8%), medical report interpretation (2,531/16,074, 15.7%), and triage guidance (2,515/16,074, 15.6%); among the 7,845 symptom consultation sessions, fever (2,719/7,845, 34.7%) and cough (2,349/7,845, 29.9%) were most common. After expert-guided optimization, the proportion of sessions rated good in overall quality increased from 65.2% (652/1,000) to 74.3% (743/1,000), and clinical appropriateness from 61.4% (614/1,000) to 87.4% (874/1,000; both P < .001); the overall turn-level deficiency rate declined from 19.4% to 6.9% (odds ratio 0.30, 95% CI 0.24–0.37; P < .001), with suboptimal clinical recommendations remaining the most frequent category.

Conclusions:

This real-world study suggests that an LLM-based health consultation assistant may complement existing pediatric care, with expert-guided optimization substantially improving response quality. However, the persistence of deficiencies highlights the need for continued monitoring and refinement to ensure clinical reliability.