JMIR Publications

ABSTRACT

Background:

Chat Generative Pre-trained Transformer (ChatGPT), a large language model, has shown good performance on physician certification exams and medical consultations. However, its performance has not been examined in languages other than English or on nursing exams.

Objective:

We aimed to evaluate the performance of ChatGPT on Japanese National Nurse Examinations.

Methods:

We evaluated the percentages of correct answers provided by ChatGPT (GPT-3.5) for all questions on the Japanese National Nurse Examinations from 2019–2023, excluding inappropriate questions and questions containing images. Inappropriate questions were pointed out by a third-party organization and announced by the government to be excluded from scoring. Specifically, these include "questions with inappropriate question difficulty" and "questions with errors in the questions or choices." These exams consists of 240 questions each year, divided into basic knowledge questions that test the basic issues of particular importance to nurses and general questions that test a wide range of specialized knowledge. The format of questions had also two types: simple-choice and situation-setup questions. Simple-choice questions are primarily knowledge-based and multiple-choice, whereas situation-setup questions entail the candidate reading a patient and family situation description, and selecting the nurse's action or patient's response. Hence, the questions were standardized using two types of prompts before requesting answers from ChatGPT. Chi-square tests were conducted to compare the percentage of correct answers for each year's exam format and specialty area related to the question. In addition, a Cochran-Armitage trend test was performed on the percentage of correct answers from 2019–2023.

Results:

The 5-year average percentage of correct answers for ChatGPT was 75.1% ± 3.0% for basic knowledge questions and 64.5% ± 5.0% for general questions. The highest percentage of correct answers on the 2019 exam was 80% for basic knowledge questions and 71.2% for general questions. ChatGPT met the passing criteria for the 2019 Japanese National Nurse Examination and was close to passing the 2020–2023 exams, with only a few more correct answers required to pass. In some areas, such as Pharmacology, Social Welfare, Related Law and Regulations, Endocrinology/Metabolism, and Skin, ChatGPT had lower percentages of correct answers, with higher percentages of correct answers in the areas of Nutrition, Pathology, Hematology, Eye, Ear Nose and Throat, Tooth and Oral, and Nursing Integration and Practice.

Conclusions:

ChatGPT only passed the 2019 Japanese National Nursing Examination during the most recent 5 years. Although it did not pass the exams from other years, it performed very close to the passing level, including on psychological, communicational, and nurse-specific questions. Clinical Trial: Not applicable.