Accepted for/Published in: JMIR Medical Education
Date Submitted: Apr 25, 2023
Open Peer Review Period: Apr 25, 2023 - Jun 20, 2023
Date Accepted: Oct 14, 2023
Date Submitted to PubMed: Oct 14, 2023
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Potential of ChatGPT as a Support Tool for Pharmacists: An Analytical Study Using the Japanese National Examination for Pharmacists
ABSTRACT
Background:
The advancement of artificial intelligence (AI), as well as machine learning, has led to its application in various industries, including healthcare. AI chatbots, such as ChatGPT developed by OpenAI, have demonstrated potential in supporting healthcare professionals by providing medical information, answering examination questions, and assisting in medical education. However, the applicability of ChatGPT in the field of pharmacy remains unexplored.
Objective:
This study aimed to evaluate ChatGPT’s ability to answer questions from the Japanese National Examination for Pharmacists (JNEP) and assess its potential as a tool for pharmacists in their daily practice.
Methods:
The question texts and answer choices from both the 107th and 108th JNEP held in February 2022 and February 2023 respectively, were input into ChatGPT (GPT-4 version). Questions requiring diagram interpretation were excluded as ChatGPT cannot process diagrams. The correct answer rates were calculated and compared with the passing criteria of each examination to evaluate ChatGPT’s performance.
Results:
For the 107th JNEP, ChatGPT correctly answered 78.2% (222/284) of the input questions, whereas, for the 108th JNEP, ChatGPT correctly answered 75.3% (217/287) of the input questions. In both examinations, the accuracy rate met the passing criteria. However, the accuracy rate for calculation questions was considerably lower (42.9%). Significant differences in ChatGPT accuracy rates were observed among the question types, fields, and whether the question was a calculation question or not.
Conclusions:
The results indicate that ChatGPT possesses the level of knowledge required of Japanese pharmacists and could support pharmacotherapy in clinical settings. Nonetheless, ChatGPT’s limitations in answering calculation questions and interpreting diagrams should be considered when using it in practice. These findings suggest that while ChatGPT can provide valuable assistance in various pharmacy scenarios, it should not be considered a complete substitute for a pharmacist’s expertise and judgment. By understanding and addressing these limitations, AI chatbots, such as ChatGPT, can become increasingly valuable resources for pharmacists and contribute to better patient care and outcomes.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.