Accepted for/Published in: JMIR Public Health and Surveillance
Date Submitted: Jun 26, 2025
Open Peer Review Period: Jun 26, 2025 - Aug 21, 2025
Date Accepted: Dec 22, 2025
(closed for review but you can still tweet)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Examining Artificial Intelligence Chatbots’ Responses in Providing Human Papillomavirus Vaccine Information for Young Adults
ABSTRACT
Background:
Background:
The growing use of artificial intelligence (AI) chatbots for seeking health-related information is concerning, as they were not originally developed for delivering medical guidance. The quality of chatbots’ responses relies heavily on their training data and is often compromised in medical contexts due to their lack of specific training data in medical literature.
Objective:
Objectives: This study examined AI chatbots responses to human papillomavirus (HPV)-related questions by analyzing structure and patterns, linguistic features, information accuracy and currency.
Methods:
Methods:
We conducted a qualitative content analysis to examine four selected AI chatbots (ChatGPT-4, Claude 3.7-Sonnet, DeepSeek-V3, and Docus [General AI Doctor]) in answering HPV vaccine questions adapted from Vaccine Conspiracy Beliefs Scale (VCBS) items and Google Trends query.
Results:
Results:
All AI chatbots cited evidence-based sources from reputable health organizations. We found no fabricated information or inaccuracies in numerical data. For complex questions, all AI chatbots appropriately deferred to healthcare professionals’ suggestions. All chatbots maintained a neutral or pro-vaccine stance, corresponding with the scientific consensus. The mean response lengths varied (word count; ChatGPT: 436.4, Claude: 188.0, DeepSeek: 510.0, Docus: 159.4), as did readability (Flesch-Kincaid Grade-Level; ChatGPT: 10.7, Claude: 13.2, DeepSeek:11.3, Docus:12.2). ChatGPT and Claude offered personalized responses, while DeepSeek and Docus lacked this. Occasionally, some responses included broken or irrelevant links and medical jargon.
Conclusions:
Conclusion: Amidst an online environment saturated with misinformation, AI chatbots have the potential to serve as alternative sources of accurate HPV-related information to conventional online platforms (websites, social media), though improvements in readability, personalization, and link accuracy are still needed. Clinical Trial: N/A
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.