JMIR Publications

ABSTRACT

Background:

Human papillomavirus (HPV) vaccine hesitancy poses significant public health challenges, particularly in Japan, where proactive vaccination recommendations were suspended between 2013 and 2021. The resulting information gap between medical institutions and vaccine-hesitant populations is exacerbated by misinformation on social media platforms. Traditional public health communication strategies cannot address individual queries simultaneously while monitoring population-level discourse.

Objective:

This study aimed to develop and evaluate a dual-purpose artificial intelligence (AI) agent system that provides verified HPV vaccine information to the public through a conversational interface while generating analytical reports for medical institutions based on user interactions and social media discourse.

Methods:

We implemented a system comprising three components: a vector database integrating documents from academic papers, Japanese government sources, news media, and social media posts; a retrieval-augmented generation chatbot using a ReAct agent architecture with iterative multitool orchestration across five specialized knowledge sources; and an automated report generation system with modules for news analysis, research synthesis, social media sentiment analysis, including stance classification and topic modeling, and user interaction pattern identification. System performance was assessed using both automated and manual evaluation protocols with a 0--5 scoring scale.

Results:

The entire system functioned as expected. For single-turn evaluation, the chatbot achieved mean scores (SD) of 4.83 (0.67) for relevance, 4.89 (0.53) for routing, 4.50 (1.29) for reference quality, 4.90 (0.62) for correctness, and 4.88 (0.54) for professional identity, with an overall mean of 4.80 (0.80). Multiturn evaluation yielded higher scores: context retention 4.94 (0.25), topic coherence 5.00 (0.00), and overall mean 4.98 (0.15), with topic-centering and identity achieving 5.00. The report generation system achieved high scores across all sections: completeness ranged from 4.00 to 5.00 (consistently perfect paper sections); correctness from 4.00 to 5.00; and helpfulness from 3.67 to 5.00. Reference validity achieved perfect scores (5.00) across all periods, with citation correctness averaging 4.33 (0.50) for news sections and 4.08 (0.66) for paper sections.

Conclusions:

This study demonstrated the feasibility of an integrated AI agent system for bidirectional HPV vaccine communication in Japan. The architecture enabled verified information delivery with source attribution while providing institutional stakeholders with a systematic public discourse analysis. This transferable framework provides a foundation for adaptation to other vaccines and multilingual public health contexts.