Accepted for/Published in: JMIR Medical Informatics
Date Submitted: Jan 16, 2024
Date Accepted: Jul 21, 2024
Assessing ChatGPT as a Medical Consulting Assistant for Chronic Hepatitis B: A Cross-Language Study of English and Chinese
ABSTRACT
Background:
Chronic hepatitis B imposes substantial economic and social burdens globally. Managing CHB involves intricate monitoring and adherence challenges, particularly in regions like China, where a high prevalence intersects with healthcare resource limitations. This study explores the potential of ChatGPT-3.5, an emerging AI assistant, to address these complexities. With notable capabilities in medical education and practice, ChatGPT-3.5's role is examined in managing CHB, particularly in regions with distinct healthcare landscapes.
Objective:
This study aims to uncover insights into ChatGPT-3.5’s potential and limitations in delivering personalized medical consulting assistance for chronic hepatitis B patients across diverse linguistic contexts.
Methods:
Questions sourced from published guidelines, online chronic hepatitis B communities, and search engines in English and Chinese were refined, translated, and compiled into 96 inquiries. These questions were independently presented to ChatGPT-3.5 in dialogues. Responses underwent evaluation by senior physicians, focusing on informativeness, emotional management, consistency across repeated inquiries and cautionary statements regarding medical advice.
Results:
Over half of the responses from ChatGPT-3.5 were deemed comprehensive. Superior performance was observed in English, particularly in informativeness and consistency across repeated queries. However, deficiencies were noted in emotional management guidance.
Conclusions:
In this study, ChatGPT demonstrates potential as a medical consulting assistant for chronic hepatitis B management. The choice of working language by ChatGPT is identified as a potential factor influencing its performance, particularly concerning the utilization of terms and jargon, which may impact the applicability of ChatGPT within specific target populations. This study highlights the significance of providing language-specific training and incorporating emotional management strategies when deploying ChatGPT for medical purposes similar to those investigated.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.