JMIR Publications

ABSTRACT

Background:

High-quality patient education for Deep Brain Stimulation (DBS) is crucial. While Large Language Models (LLMs) offer new information channels, their comparative quality in this high-risk neurosurgical field remains unknown, particularly regarding the complex trade-offs between scientific accuracy, patient comprehensibility, and empathy.

Objective:

This study conducted a multidimensional comparative evaluation of three mainstream LLMs (ChatGPT 5.0, Gemini 2.5, and DeepSeek 3.2) to assess their performance, safety, and suitability for DBS patient education.

Methods:

We constructed a standardized set of 50 Frequently Asked Questions (FAQs) derived from official guidelines and patient forums. Each LLM generated independent responses, yielding 150 total texts. A blinded review panel of 10 experts (8 neurosurgeons and 2 health communication specialists) evaluated the responses using a 5-point Likert scale across five dimensions. Subjective scores were analyzed using Linear Mixed-Effects Models (LMMs) to account for repeated measures; objective Flesch-Kincaid readability tests used the same architecture.

Results:

Significant differences emerged across ALL five dimensions (P < .001). ChatGPT achieved the highest Accuracy (Mean = 4.68, 95% CI [4.52, 4.84]), significantly outperforming Gemini (Mean = 4.20, 95% CI [4.04, 4.36]; Cohen's d = -1.21). DeepSeek achieved the highest Relevance (Mean = 4.78, 95% CI [4.66, 4.90]), while Gemini scored lowest (Mean = 4.31, 95% CI [4.19, 4.43]; d = -1.19 vs. DeepSeek). ChatGPT scored highest in Completeness (Mean = 4.90, 95% CI [4.77, 5.00]) but was rated lowest for subjective Comprehensibility (Mean = 3.98, 95% CI [3.71, 4.25]) and Humanistic Care (Mean = 3.40, 95% CI [3.17, 3.63]). Conversely, DeepSeek and Gemini demonstrated superior empathy; DeepSeek achieved the highest Humanistic Care score (Mean = 4.67, 95% CI [4.45, 4.90]; P < .001 vs. ChatGPT). Objective Flesch-Kincaid analysis corroborated these findings (P < .001): Gemini provided the easiest readability (Grade Level = 10.08, 95% CI [9.49, 10.67]), equivalent to a 10th-grade (high-school) reading level. DeepSeek followed at a high-school graduate level (Grade Level = 12.04, 95% CI [11.46, 12.63]). ChatGPT's responses required college-level reading proficiency (Grade Level = 13.51, 95% CI [12.93, 14.10]), posing a significant barrier for patients.

Conclusions:

LLMs are NOT interchangeable tools for patient education, with significant differences emerging across all dimensions including scientific accuracy itself. A critical behavioral trade-off exists: ChatGPT provides accurate and complete but complex and "cold" answers, making it unsuitable for direct patient queries. DeepSeek uniquely balances high relevance and empathy with near-comparable accuracy. Gemini provides superior readability but shows consistent deficits in accuracy, relevance, and completeness, rendering its unsupervised use for patients clinically risky. We propose a tiered implementation framework—utilizing high-accuracy models like ChatGPT as "clinician drafting assistants" and balanced empathetic models like DeepSeek as verified "initial patient resources"—to safely integrate AI into DBS patient care.