JMIR Publications

ABSTRACT

Background:

The disease burden of breast cancer in China is increasing. Guiding people to obtain correct information about breast cancer and improving the public's health literacy are crucial for the early detection and timely treatment of breast cancer. Large Language Models (LLMs) are currently popular sources of health information. However, the accuracy and practicality of the breast cancer-related information provided by LLMs have not been evaluated yet.

Objective:

This study aims to evaluate and compare the accuracy and practicality of responses to breast cancer-related questions from two LLMs, ChatGPT and ERNIE Bot (EB).

Methods:

The questions asked to the LLMs consist of patient questionnaires and expert questionnaires, each containing 15 questions. ChatGPT was asked in Chinese and English, recorded as ChatGPT-English (ChatGPT-E) and ChatGPT-Chinese (ChatGPT-C) respectively, and EB was asked in Chinese. The accuracy and practicality of each inquiry's results are rated by the breast cancer multidisciplinary treatment team using Likert scales.

Results:

Overall, whether in patient questionnaire or expert questionnaire, the accuracy and practicality of responses from ChatGPT-E are significantly higher than those from ChatGPT-C and EB. However, the responses of LLMs are relatively generalised, leading to lower accuracy and practicality on expert questionnaire compared to patient questionnaire. Additionally, there are issues with the lack of supporting evidence as well as potential ethical risks in the responses of LLMs.

Conclusions:

Currently, compared to other LLMs, ChatGPT-E has demonstrated greater potential for application in educating Chinese breast cancer patients, and can be serving as an effective tool for them to obtain health information. However, for breast cancer doctors, these LLMs are still not suitable for assisting in clinical diagnosis and treatment activities. Additionally, the data security, ethical, and legal risks brought by LLMs in clinical practice cannot be ignored. In the future, further research is needed to determine the true efficacy of LLMs in the application of clinical scenarios related to breast cancer in China.