JMIR Publications

ABSTRACT

Background:

The advancement of artificial intelligence (AI), as well as machine learning, has led to its application in various industries, including healthcare. AI chatbots, such as ChatGPT developed by OpenAI, have demonstrated potential in supporting healthcare professionals by providing medical information, answering examination questions, and assisting in medical education. However, the applicability of ChatGPT in the field of pharmacy remains unexplored.

Objective:

This study aimed to evaluate ChatGPT’s ability to answer questions from the Japanese National Examination for Pharmacists (JNEP) and assess its potential as a tool for pharmacists in their daily practice.

Methods:

The question texts and answer choices from both the 107th and 108th JNEP held in February 2022 and February 2023 respectively, were input into ChatGPT (GPT-4 version). Questions requiring diagram interpretation were excluded as ChatGPT cannot process diagrams. The correct answer rates were calculated and compared with the passing criteria of each examination to evaluate ChatGPT’s performance.

Results:

For the 107th JNEP, ChatGPT correctly answered 78.2% (222/284) of the input questions, whereas, for the 108th JNEP, ChatGPT correctly answered 75.3% (217/287) of the input questions. In both examinations, the accuracy rate met the passing criteria. However, the accuracy rate for calculation questions was considerably lower (42.9%). Significant differences in ChatGPT accuracy rates were observed among the question types, fields, and whether the question was a calculation question or not.

Conclusions:

The results indicate that ChatGPT possesses the level of knowledge required of Japanese pharmacists and could support pharmacotherapy in clinical settings. Nonetheless, ChatGPT’s limitations in answering calculation questions and interpreting diagrams should be considered when using it in practice. These findings suggest that while ChatGPT can provide valuable assistance in various pharmacy scenarios, it should not be considered a complete substitute for a pharmacist’s expertise and judgment. By understanding and addressing these limitations, AI chatbots, such as ChatGPT, can become increasingly valuable resources for pharmacists and contribute to better patient care and outcomes.