JMIR Publications

ABSTRACT

Background:

The creation of large language models (LLMs) is a significant step in the evolution of artificial intelligence (AI). The United States and China standardise the theoretical and practical knowledge of nurses through the National Council Licensure Examination for Registered Nurses (NCLEX-RN) and the National Nursing Licensure Examination (NNLE), respectively. However, ineffective nursing education instruments are lacking.

Objective:

Examine how well LLMs respond to U.S. and Chinese multiple-choice questions (MCQs) in the nursing licensure exams. To determine whether LLMs can be utilised as multilingual learning assistance for nursing.

Methods:

First, we compiled 150 NCLEX-RN practise questions, 240 NNLE theoretical MCQs, and 240 NNLE practical MCQs. Then, the translation function of ChatGPT 3.5 is utilised to translate NCLEX-RN questions from English to Chinese and NNLE questions from Chinese to English. Finally, the original version and the translated version of the MCQs are inputted into ChatGPT 4.0, ChatGPT 3.5, and Google Bard, respectively.

Results:

The accuracy rates of ChatGPT4.0 for NCLEX-RN practise questions and Chinese-translated NCLEX-RN practise questions were 88.7% and 79.3%, respectively. Despite the statistical significance of the difference(P=.03), the correct rate was generally satisfactory. 71.9% of NNLE Theoretical MCQs and 69.1% of NNLE Practical MCQs were correctly answered by ChatGPT4.0. The accuracy of ChatGPT4.0 in processing NNLE Theoretical MCQs and NNLE Practical MCQs translated into English was 71.5%(P=.92) and 67.8%(P=.77), respectively, and there was no statistically significant difference between the results of text input in different languages. ChatGPT3.5(NCLEX-RN’s P=.003,NNLE Theoretical P<.001, NNLE Practical P=.12) and Google Bard(NCLEX-RN’s P<.000,NNLE Theoretical P<.001, NNLE Practical P<.001) have lower accuracy rates for nursing-related MCQs than ChatGPT4.0 in English input. English accuracy was higher when compared to ChatGPT3.5's Chinese input, and the difference was statistically significant(NCLEX-RN’s P=.02, NNLE Practical P=.02).

Conclusions:

ChatGPT 4.0 has a high level of accuracy on both the US and Chinese nursing exams' MCQs and provides a thorough explanation of the questions' knowledge points. ChatGPT can be used as an effective auxiliary tool for nursing knowledge learning. Clinical Trial: Not applicable.