JMIR Publications

ABSTRACT

Background:

Artificial intelligence chatbots, including those used in the field of ultrasound, are increasingly used to answer medical questions. However, there are many chatbot models with varying performance, and the model's performance is also affected by multiple factors, such as language environment, question type, and topic.

Objective:

This study aimed to evaluate the performance of the ChatGPT and the ERNIE Bot in answering questions related to ultrasound medical examinations, providing a reference for users and developers.

Methods:

In this study, we collected actual examination papers from the field of ultrasound medicine and strictly selected 554 questions, including single-choice, multiple-choice, true or false questions, noun explanations, and short answers. The topics included basic knowledge, ultrasound examination, diagnosis, diseases and etiology, case analysis, and ultrasound signs. The questions were asked in both English and Chinese. Objective questions were evaluated based on the correct response rate, and subjective questions were evaluated by a doctor with more than 20 years of work experience and proficiency in both Chinese and English using a Likert scale. The data were imported into Excel for comparison analysis.

Results:

Of the 554 questions included in this study, single-choice questions accounted for the greatest proportion (64%), followed by short answers (12%) and noun explanations (11%), and the remaining questions were multiple-choice and true-false questions. The accuracy rates of the objective questions were ranked in the following order: true or false questions (60%-80%), single-choice questions (57.34%-62.99%), and multiple-choice questions (8.33%-39.58%). The acceptability rate of short answers to subjective questions was 65.22%~75.36%, which was slightly greater than that of noun interpretations (47.62%~61.9%). In terms of the performance comparison between models, ERNIE Bot performed slightly better than ChatGPT in several aspects. Both models showed a decline in performance when the examination questions were translated into English, but the decline was less pronounced in the ERNIE Bot. In terms of topic categories, the model performed better in terms of basic knowledge, ultrasound examination methods, diseases and etiology than in terms of ultrasound signs and ultrasound diagnosis.

Conclusions:

In this cross-sectional study, chatbots can provide valuable answers to ultrasound examination questions, but there are performance differences between models, and the performance of the models is closely related to the input language, question type and topic. Overall, the answers of the ERNIE Bot are superior to those of the ChatGPT in many aspects. As users or developers, it is necessary to have a deep understanding of the performance characteristics of the models and select different models for different questions and language environments to fully utilize the value of chatbots and continuously optimize and improve chatbot performance. Clinical Trial: NONE