JMIR Publications

ABSTRACT

Background:

Systemic Lupus Erythematosus (SLE) is a complex, fluctuating disease, creating a continuous need for reliable patient information. A prior study concluded that SLE patients often turn to the internet, including AI chatbots, for information regarding SLE. The rise of AI chatbots as a primary information source presents a critical challenge regarding accuracy.

Objective:

This study aims to evaluate the performance of the latest generation of AI chatbots (ChatGPT-4o, DeepSeek V.3, and Gemini 2.5 Flash) when it comes to answering Frequently Asked Questions about SLE.

Methods:

Twenty-two frequently asked questions (FAQs) about SLE, in Bahasa Indonesia, were posed to each chatbot. Responses were independently and blindly evaluated for accuracy by five clinical immunologists using a 4-point Likert scale. Readability was assessed using the Flesch Reading Ease Indonesian (FRES score) formula. Statistical comparisons for accuracy and readability were performed (Kruskal-Wallis/ANOVA), and Spearman's Rho was used to correlate accuracy, readability, and word count.

Results:

Gemini 2.5 Flash demonstrated the highest accuracy with a mean (SD) of 1.25 (0.53), significantly outperforming DeepSeek V.3 (1.48 ± 0.63) and ChatGPT-4o (1.71 ± 0.61) (p < .001). Gemini 2.5 Flash also significantly scored better across all four subdomains. Readability for all three chatbots was low (median FRES score: 42.22–46.66). Gemini 2.5 Flash produced the longest responses (8509 words total), followed by DeepSeek V.3 (5410 words) and ChatGPT-4o (3632 words). A weak but significant positive correlation was found between word count and lower accuracy (ρ = +0.375, p = .002).

Conclusions:

Gemini 2.5 Flash provided the most accurate responses for SLE-related questions, demonstrating consistent performance across all domains. However, its clinical utility, along with that of ChatGPT-4o and DeepSeek V.3, is severely limited by low readability scores (FRES < 50), making it unsuitable for general patient use. This highlights a critical "blind spot" where clinical accuracy, as rated by experts, does not equate to patient accessibility. Thus, further research is required to assess accuracy and readability of Chatbot AIs across different medical fields and topics.