JMIR Publications

ABSTRACT

Background:

People living with dementia (PLWD) or other memory problems, and their caregivers increasingly rely on the Internet to find information about their condition, available resources and services. The recent advancements in Large Language Models (LLMs), such as ChatGPT, provide a new alternative to the more traditional Web search engines like Google.

Objective:

This study compares the quality of the results of ChatGPT and Google for a collection of PLWD-related queries.

Methods:

A set of 30 informational and 30 service delivery (transactional) PLWD-related queries were selected and submitted to both Google and ChatGPT. Three domain experts assessed the results for their currency of the information, reliability of the source, objectivity, relevance to the query, and similarity of their response. The readability of the results was also analyzed. Interrater reliability coefficients were calculated for all outcomes.

Results:

Google had superior currency, as the results of ChatGPT may be out-of-date and often do not specify a validity timestamp. Google also had higher reliability, because ChatGPT rarely includes the source of a result. ChatGPT’s results were evaluated to be more objective, because Google sometimes returns results based on commercial entities. ChatGPT has significantly higher response relevance, because Google often drew upon sources that were referral services for dementia care or service providers themselves. The readability was low for both platforms, especially for ChatGPT (mean grade level 12.17 vs. 9.86 for Google). The similarity between the content of ChatGPT and Google responses were rated as high for 13(21.7%) responses, medium for 16(26.7%) responses, and low for 31(51.6%) responses.

Conclusions:

Both Google and ChatGPT have strengths and weaknesses. Google more often provides a date for and a known reliable source of the response compared to ChatGPT, whereas ChatGPT supplies more relevant responses to queries . The readability scores for both indicate responses are often not appropriate for persons with low health literacy skills. In the future, the addition of both the source and date of the health-related information and availability in other languages may increase the value of these platforms for both non-medical and medical professionals.