JMIR Publications

ABSTRACT

Background:

The integration of artificial intelligence (AI), notably Chat Generative Pre-trained Transformer (ChatGPT), into medical education, has shown promising results in various medical fields. Nevertheless, its efficacy in Traditional Chinese Medicine (TCM) examinations remains understudied.

Objective:

This study aims to (1) assess the performance of ChatGPT on the TCM licensing examination in Taiwan, and (2) explore its potential as a learning tool and its understanding of TCM principles.

Methods:

We used the GPT-4 model to respond to 480 questions from the 2022 TCM licensing examination. This study compared the performance of the model against that of licensed TCM doctors using two approaches, namely direct answer selection and provision of explanations before answer selection. The accuracy and consistency of AI-generated responses were analyzed. Moreover, a breakdown of question characteristics was performed based on the cognitive level, depth of knowledge, types of questions, vignette style, and polarity of questions.

Results:

ChatGPT achieved an overall accuracy of 43.9%, which was lower than that of two human participants (70% and 78.4%). The analysis did not reveal a significant correlation between the accuracy of the model and the characteristics of the questions. An in-depth examination indicated that errors predominantly resulted from misunderstanding of TCM concepts (55.3%), emphasizing the limitations of the model with regard to TCM knowledge base and reasoning capability.

Conclusions:

While ChatGPT shows promise as an educational tool, its current performance on TCM licensing examinations is lacking. This highlights the need for enhancing AI models with specialized TCM training and suggests a cautious approach to utilize AI for TCM education. Future research should focus on model improvement and the development of tailored educational applications to support TCM learning.