Previously submitted to: JMIR Formative Research (no longer under consideration since Jul 06, 2025)
Date Submitted: Nov 12, 2024
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
The performance of large language models in Helicobacter pylori related medical counseling within a Chinese-language context: A comparative analysis
ABSTRACT
Background:
Helicobacter pylori (H. pylori) infection is a global health issue, leading to a growing demand for medical counseling. Large Language Models (LLMs) have the potential to serve as valuable auxiliary tools for medical counseling. However, their performance in terms of accuracy, relevance, completeness, clarity, and reliability in providing H. pylori related medical counseling within a Chinese-language context remains unclear.
Objective:
This study aimed to evaluate the effectiveness of three LLMs: ChatGPT 3.5 turbo（OpenAI）, Kimi（Moonshot AI）, and Ernie Bot 3.5(Baidu, Inc), in providing H. pylori related medical counseling in a Chinese context.
Methods:
A total of 20 H. pylori related questions were included covering the following domains: definition and symptoms, diagnosis, treatment and prevention. Each question was being asked three times in Chinese to each LLM. An evaluation team of 3 physicians assessed the responses using a Likert scale across 5 dimensions: accuracy, relevance, completeness, clarity, and reliability, and came up with an overall assessment.
Results:
A total of 20 H. pylori related questions were included covering the following domains: definition and symptoms, diagnosis, treatment and prevention. Each question was being asked three times in Chinese to each LLM. An evaluation team of 3 physicians assessed the responses using a Likert scale across 5 dimensions: accuracy, relevance, completeness, clarity, and reliability, and came up with an overall assessment.
Conclusions:
This study is the first to evaluate the effectiveness of various LLMs in H. pylori related medical counseling in a real-world setting. All the interactions were conducted in Chinese, not English, demanding a higher level of linguistic comprehension. The study showed that while the LLMs generally performed acceptably in accuracy, relevance and completeness, their clarity and reliability were less satisfactory. Kimi and Ernie Bot, both developed by Chinese companies, outperformed ChatGPT in some aspects of medical counseling in Chinese language. With the guidance of professionals, LLMs can serve as potential aids for medical counseling.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.