JMIR Publications

ABSTRACT

Background:

Recent surveys indicate much of the population is willing to use generative artificial intelligence (AI) for health-related inquiries. Most research has investigated the accuracy of responses produced by AI chatbots, while others have shown that both physicians and consumers overwhelmingly prefer responses created by chatbots compared to human physicians.

Objective:

To characterize and compare the emotional content of responses from physicians and two AI chatbots (ChatGPT and Google Gemini). Reading level difficulty of responses was also compared, as was chatbot usage of disclaimers stating their responses do not qualify as medical advice.

Methods:

A public, patient-deidentified telehealth website was used to create a dataset of 100 questions asked to website physicians. Chatbot responses were generated by prompting original questions between May 18-19, 2025. Emotional content classification of each sentence was assigned by two coders using the same predefined codebook, reviewed for agreement. Emotions were then ranked as primary, secondary, and tertiary for each response based on the percentage of the response that was classified as each emotion and comparisons were made via multinomial logistic regression with physician responses as the reference group. Word count, Flesch Reading Ease (FRE), and Flesch-Kincaid Grade Level (FKGL) readability scores were calculated and assessed via ANOVA followed by Tukey’s HSD test. Disclaimer usage was calculated as the proportion of responses containing some form of a disclaimer statement and was compared between chatbots via chi-square test.

Results:

Primary emotion responses were almost completely neutral/rational, except for one response from each chatbot where anger was primary. For secondary emotions, the odds of hope were 80.28% lower (95% CI, 37.71%-93.76%) for ChatGPT while the odds of fear were 3.29 times higher (95% CI, 1.44-7.49) for Gemini. For tertiary emotions, the odds of compassion were 1.94 times higher (95% CI, 1.06-3.54) and the odds of no tertiary emotion were 84.33% lower (95% CI, 64.72%-93.04%) for Gemini. On average, Gemini used 889.1 (SD 305.7) words per response, ChatGPT used 476.5 (SD 109.5) words per response, and physicians used 193.5 (SD 113.6) words per response. Gemini had the lowest average FRE score at 39.9 (SD 8.8), followed by ChatGPT at 45.8 (SD 12.8), while physicians had the highest average score at 51.9 (SD 13.6). Gemini had the highest average FGKL score of 11.3 (SD 1.5), followed by ChatGPT with an average score of 9.9 (SD 1.9), and physicians had an average score of 9.2 (SD 2.4). Gemini was significantly more likely to use a disclaimer than ChatGPT (Χ_1^2=49.22, P<.001).

Conclusions:

Chatbot responses were significantly longer and more difficult to than physician responses. Chatbot responses were also more likely to contain more emotions and more varied emotions than physician responses. Qualitatively, chatbot responses were more varied in the way they were presented as well as in the breadth of the emotions themselves. This higher quality emotional content could be used to inform more emotionally connective physician responses to patient message queries.