Assessing the quality of ChatGPT responses to dementia caregivers’ questions
ABSTRACT
Background:
ChatGPT became publicly available in November 2022, and both patients and caregivers are using such artificial intelligence (AI) to answer their health questions. Studies have examined ChatGPT responses to patients’ questions about heart disease, cirrhosis, and bariatric surgery, but to our knowledge, none have looked at the quality of ChatGPT responses to topics relevant to dementia caregivers. AI such as ChatGPT holds great promise to improve dementia patients’ and caregivers’ quality of life by providing high-quality responses to their questions about typical dementia behaviors. So far, however, evidence on the quality of such ChatGPT responses is limited.
Objective:
This pilot study examines the potential of AI with CharGPT to provide high-quality information that may enhance dementia care and patient-caregiver education.
Methods:
Our interprofessional team used a formal rating scale to evaluate ChatGPT responses to real-world questions posed by dementia caregivers. For examination, we selected 60 posts by dementia caregivers from Reddit, a popular social media platform. These posts were verified by 3 interdisciplinary dementia clinicians as representing dementia caregivers’ wants and needs for information in the areas of memory loss and confusion, driving, and aggression.
Results:
ChatGPT quality scores ranged from 3 to 5 (highest quality); overall, 26 (43%) of the 60 responses received 5 points; 21 (35%), 4 points; and 13 (21.7%), 3 points (Table 2), suggesting high quality.
Conclusions:
ChatGPT provided high-quality responses to complex questions posted by dementia caregivers, but it did have limitations. For example, ChatGPT was unable to anticipate future problems that a human professional might recognize and address in a clinical encounter. At other times, ChatGPT recommended a strategy that the caregiver had already explicitly tried. This pilot study suggests the potential of AI to provide high-quality information to enhance dementia care and patient-caregiver education in tandem with information provided by licensed health care professionals. Evaluating the quality of responses is necessary to ensure that older adults can make informed decisions; ChatGPT has the potential to transform the field of medicine by shaping how caregivers receive health information. Clinical Trial: na
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.