JMIR Publications

ABSTRACT

Background:

Patients with chronic kidney disease (CKD) on dialysis face complex self-management challenges that generate persistent information needs traditional healthcare fails to address. Retrieval-augmented generation (RAG) improves large language model accuracy, but whether personalized RAG outperforms general RAG in reducing these needs remains unclear.

Objective:

This study assessed whether personalized constraint RAG (C-RAG) more effectively reduces information needs among dialysis patients compared to open-domain general RAG (O-RAG) and standard care, and identified predictors of chatbot engagement.

Methods:

A two-phase sequential mixed-methods study was conducted. Phase 1 evaluated four large language models (Claude 3.5 Sonnet, Gemini 1.5 Flash, CLOVA X, ChatGPT-4o) using 60 patient-generated questions assessed by 15 clinical experts. Phase 2 was a three-arm parallel pilot randomized controlled trial (October–December 2025) across three tertiary hospitals in Seoul, South Korea. Two chatbot variants were developed using GPT-4o: C-RAG, incorporating patient-specific clinical parameters for personalized responses, and O-RAG, using the same knowledge base without personalization. Participants were randomized 1:1:1 to C-RAG, O-RAG, or standard care. The primary outcome was change in information needs across medication, dietary, and diagnostic domains (5-point Likert scales, baseline and 4 weeks), analyzed using ANCOVA. Semi-structured interviews with nine participants provided mechanistic insights.

Results:

Phase 1 identified ChatGPT-4o as the optimal model, achieving the highest clinical accuracy (mean 3.60), safety (mean 3.67), and readability (Flesch-Kincaid Grade Level 9.8 vs 13.2–14.5 for other models). In Phase 2, 45 participants were enrolled; 42 completed the study (42/45, 93% retention; 14 per group). ANCOVA revealed significant group differences in medication (F2,38=4.574, P=.017) and dietary information needs (F2,38=4.232, P=.022). Bonferroni-corrected pairwise comparisons showed C-RAG had significantly lower information needs than O-RAG for medication (observed means 2.93 vs 3.86; Cohen d=-0.86, P=.017) and dietary domains (observed means 2.93 vs 3.86; Cohen d=-0.86, P=.018). No significant differences were found between either RAG group and the control group. Secondary outcomes showed no significant group differences. Dietary management was the predominant query topic (205/691, 29.3%). Participants dialyzed for 1–3 years showed higher engagement (9/10, 90% vs 5/18, 27.8%; Fisher exact P=.006). Qualitative analysis identified five themes centered on disease-specific access, dietary management as the primary need, and dialysis duration as a moderator of utility.

Conclusions:

Personalized RAG significantly outperforms general RAG with large effect sizes (Cohen d=-0.86) substantially exceeding digital health benchmarks. Medium-to-large effects against standard care (Cohen d=-0.54 to -0.64) support a fully powered confirmatory trial, with an estimated 50–55 participants per group needed to evaluate long-term clinical outcomes. Clinical Trial: KCT0011656; https://cris.nih.go.kr/cris/search/detailSearch.do?seq=32626