Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Oct 12, 2024
Date Accepted: Apr 14, 2025
Application of artificial intelligence chatbot in responding to asynchronous text-based messages from oncology patients: A comparative study.
ABSTRACT
Background:
Telemedicine, incorporating artificial intelligence such as chatbots, offers potential for enhancing healthcare delivery. However, the efficacy of chatbots compared to human physicians in clinical settings remains under-explored, particularly in complex scenarios involving tumor patients.
Objective:
This study aims to explore the performance of the chatbot in handling different clinical scenarios within telemedicine by comparing its responses to those of physicians.
Methods:
We collected 4,257 telemedicine records from tumor patients and categorized them into two clinical scenarios: medical consultation and medical decision-making. The responses from the chatbot and physicians were evaluated by a medical review panel and a patient panel.
Results:
In the medical consultation scenario, 97.34% of the chatbot’s responses were deemed complete compared to 93.61% of physicians’ responses, with no differences in accuracy or safety. In the medical decision-making scenario, 3.12% of the chatbot’s responses and 2.01% of physicians’ responses were identified as potentially leading to adverse outcomes. Regarding reliability, 93.29% of physicians’ responses were rated as "good" or higher, compared to 50.71% chatbot responses.
Conclusions:
In conclusion, while the chatbot performs well in medical consultations, it is less reliable in medical decision-making, requiring further improvement.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.