Accepted for/Published in: JMIR Medical Education
Date Submitted: Apr 24, 2023
Open Peer Review Period: Apr 23, 2023 - Jun 18, 2023
Date Accepted: Jul 25, 2023
Date Submitted to PubMed: Aug 10, 2023
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Evaluating ChatGPT's Performance in Medication-Related Consultation Questions: A Study Utilizing Real-World Data
ABSTRACT
Background:
With a strong ability in performing natural tasks and an easy-to-use interface, ChatGPT received a lot of attention since OpenAI released it to the public.
Objective:
A prospective analysis is needed to examine the accuracy and appropriateness of the medication consultation responses generated by ChatGPT.
Methods:
A prospective cross-sectional study was conducted by the pharmacy department of a medical center in Taiwan. The test dataset included the retrospective collection of medication consultation questions from February 1, 2023, to February 28, 2023, as well as common questions regarding drug-herb interactions. Two different sets of questions were tested: real-world medication consultation questions and common questions about interactions between traditional Chinese and Western medicines. For each ChatGPT response, the appropriateness was evaluated by two experienced pharmacists. In cases where the opinions of the two pharmacists were inconsistent, a third pharmacist would make the final determination.
Results:
From 293 real-world medication consultation questions, a random selection of 80 was used to compare ChatGPT's performance. ChatGPT demonstrated a higher appropriateness rate for answering public medication consultation questions compared to questions presented to healthcare providers in a hospital setting (61% vs 39%, p=.011).
Conclusions:
The results of this study suggest that ChatGPT may be used to answer basic medication consultation questions. We analyzed the erroneous information and identified potential medical harms associated with certain questions, which is worthy of our attention.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.