Previously submitted to: JMIR AI (no longer under consideration since Mar 15, 2025)
Date Submitted: Feb 4, 2025
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Examination of the Quality and Readability of Chatbot Responses to Patient Questions: A Synthesis of Recent Studies
ABSTRACT
Background:
Patient use of chatbots to obtain medical information has been anticipated with both optimism and pessimism. The simplicity of asking questions and receiving immediate answers has prompted investigators to examine the quality and readability of chatbot responses. We sought to review the current results at this nascent stage in chatbot development.
Objective:
To evaluate current data.
Methods:
We searched multiple databases to identify studies that evaluated response quality using the DISCERN instrument; designed to assess written material intended for patients. From these studies, we extracted the DISCERN scores, the number of words used in questions, the number of questions asked, the number of evaluators, and, if recorded, the readability of the responses. We also examined a measure of the rank of the journals in which the studies were published. We combined these parameters in a multiple linear regression model to determine potential associations with response quality.
Results:
We identified 32 studies that conducted 57 tests using multiple chatbots. The average number of words in chatbot prompts ranged from 6 to 41, and the number of questions ranged from 3 to 119. As the response quality increased, readability decreased. Forty-two percent of tests produced average responses ranked as “good” or higher, and only one response test was below college-level readability. An increased DISCERN score was associated with increased prompt words and questions in simple linear regression. In a multiple linear regression model, higher DISCERN scores were associated with the number of questions, three or more evaluators, inversely with the journal rank, but not with the number of prompt words.
Conclusions:
The variable quality and poor readability of chatbot responses to patient questions reinforces pessimism about their role. However, the principles of prompt engineering (the art of asking questions) have yet to be rigorously applied. Therefore, we remain optimistic that response quality and readability will improve.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.