Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Jun 8, 2023
Open Peer Review Period: Jun 8, 2023 - Aug 3, 2023
Date Accepted: Nov 16, 2023
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Feasibility of ChatGPT 3.5/4.0 as a Source of COVID-19 Information: A Comprehensive Comparative Analysis of Generative and Authoritative Information
ABSTRACT
Background:
he COVID-19 pandemic, caused by the SARS-CoV-2 virus, has necessitated reliable and authoritative information for public guidance. The World Health Organization (WHO) has been a primary source of such information, disseminating it through a Q&A format on its official website. Concurrently, ChatGPT 3.5/4.0, a deep learning-based natural language generation (NLG) system, has shown potential in generating diverse text types based on user input.
Objective:
This study evaluates the accuracy of COVID-19 information generated by ChatGPT 3.5/4.0, assessing its potential as a supplementary public information source during the pandemic.
Methods:
We extracted 487 COVID-19-related questions from the WHO's official website and used ChatGPT 3.5/4.0 to generate corresponding answers. These were compared with the official WHO responses. Two clinical experts scored the generated answers (0-5) across four dimensions (accuracy, comprehensiveness, relevance, clarity), using the WHO responses as a reference. Additionally, we used the Bert model to generate similarity scores (0-1) between the generated and official answers, providing a dual validation mechanism.
Results:
The average scores for ChatGPT 3.5-generated answers were 3.47 (accuracy), 3.89 (comprehensiveness), 4.09 (relevance), and 3.49 (clarity). For ChatGPT 4.0, the scores were 4.15, 4.47, 4.56, and 4.09, respectively. All differences were statistically significant (P<.05). Bert model verification showed average similarity scores of 0.83±0.07 (ChatGPT 3.5) and 0.85±0.07 (ChatGPT 4.0) compared with the official WHO answers.
Conclusions:
ChatGPT 3.5/4.0 can generate accurate and relevant COVID-19 information to a certain extent. However, compared with official WHO responses, gaps and deficiencies exist. Thus, users of ChatGPT 3.5/4.0 should also reference other reliable information sources to mitigate potential misinformation risks. Notably, ChatGPT 4.0 outperformed ChatGPT 3.5 across all evaluated dimensions, a finding corroborated by Bert model validation.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.