JMIR Publications

ABSTRACT

Background:

As large language model (LLM)-based chatbots such as ChatGPT grow in popularity, it is essential to understand their role in delivering online health information compared to other resources. These chatbots often generate inaccurate content, posing potential safety risks. This motivates the need to examine how users perceive and act on health information provided by LLM-based chatbots.

Objective:

This study investigates the patterns, perceptions, and actions of users seeking health information online, including LLM-based chatbots. The relationships between online health information-seeking behaviors and important sociodemographic characteristics are examined as well.

Methods:

An online survey of crowd workers was conducted via Prolific. The questionnaire covered sociodemographic information, trust in healthcare providers, eHealth literacy, artificial intelligence (AI) attitudes, chronic health condition status, online health information source types, perceptions, and actions, such as cross-checking or adherence. Quantitative and qualitative analyses were applied.

Results:

Most participants consulted search engines (98.0%, 291/297) and health-related websites (68.4%, 203/297) for their health information, while 21.2% (63/297) used LLM-based chatbots with ChatGPT and Microsoft Copilot being the most popular. The majority of participants (90.2%, 268/297) sought information on health conditions, with fewer seeking advice on medication (60.3%, 179/297), treatments (46.1%, 137/297), and self-diagnosis (23.2%, 62/297). Perceived information quality and trust varied little across source types. The preferred source for validating information from the Internet was consulting healthcare professionals (30.3%, 40/132) while only a very small percentage of participants (2.3%, 5/214) consulted AI tools to cross-check information from search engines and health-related websites. For information obtained from LLM-based chatbots, 19.4% of participants (12/63) cross-checked the information while 48.4% of participants (30/63) followed the advice. Both of these rates were lower than information from search engines, health-related websites, forums, or social media. Furthermore, use of LLM-based chatbots for health information was negatively correlated with age (ρ=-0.16 P=.00). On the other hand, attitudes surrounding AI for medicine had significant positive correlations with the number of source types consulted for health advice (ρ=0.14, P=.01), use of LLM-based chatbots for health information (ρ=0.31, P=.00), and number of health topics searched (ρ=0.19, P=.00).

Conclusions:

While traditional online sources remain dominant, LLM-based chatbots are emerging as a resource for health information for some users, specifically those who are younger and have a higher trust in AI. The perceived quality and trust of health information varied little across source types. However, the adherence of health information from LLM-based chatbots seemed more cautious compared to search engines or health-related websites. As LLMs continue to evolve, enhancing their accuracy and transparency will be essential in mitigating any potential risks by supporting responsible information seeking while maximizing the potential of AI in health contexts.