Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: May 2, 2025
Date Accepted: Aug 6, 2025
Effectiveness, usability, and acceptability of a ChatGPT with Retrieval-Augmented Generation (the SIV-ChatGPT) in increasing seasonal influenza vaccination uptake among older adults: A quasi-experimental study
ABSTRACT
Background:
Older adults are more vulnerable to severe consequences caused by seasonal influenza. Although seasonal influenza vaccination (SIV) is effective and free vaccines are available, the SIV uptake rate remained inadequate among people aged 65 years or above in Hong Kong, China. These was a lack of studies evaluating a ChatGPT in promoting vaccination uptake among older adults.
Objective:
The present study was to evaluate the effectiveness of a ChatGPT with Retrieval-Augmented Generation (RAG) in increasing SIV uptake among older adults over a 3-month study period in Hong Kong, China. Participants in an ongoing observational cohort study conducted in the same period served as the comparison group.
Methods:
A quasi-experimental study was conducted between November 2024 and April 2025. Participants were: 1) aged ≥65 years; 2) possession of a Hong Kong ID; 3) able to speak and comprehend Cantonese; 4) smartphone users; and 5) no SIV uptake for the approaching flu season. Those with diagnosis of cognitive impairment or dementia, either blindness or deafness or known contraindications to the SIV were excluded. Participants were recruited through random telephone calls. There were 45 and 55 participants in the SIV-ChatGPT group and the comparison group, respectively. All participants completed follow-up surveys at T1 (one month after baseline survey, for SIV-ChatGPT group only) and T2 (three months after baseline survey, for both groups). Participants in the SIV-ChatGPT group gained access to the SIV-ChatGPT, in the format of a web-based Application, after completion of the baseline survey. They could use the SIV-ChatGPT repeatedly throughout a one-month intervention period, and were free to spend as much time as they wanted with the SIV-ChatGPT. Intention-to-treat analysis was used for outcome analyses.
Results:
At T2, the SIV uptake rate was higher in the SIV-ChatGPT group than the comparison group (33% [15/45] versus 14.3% [8/55]; adjusted odds ratios [AOR]: 2.72, 95%CI: 1.01, 7.35, p=0.048). All participants were able to provide receipts to validate their SIV uptake. In the SIV-ChatGPT group, 40.5% (15/37) of participants who used the SIV-ChatGPT at least once reported a SIV uptake at T2, which was significantly higher than non-users (0/8, 0%; p=0.04). Among 37 users of the SIV-ChatGPT, the mean score of the System Usability Scale (SUS) was 67.1 (standard deviation [SD]: 14.9). Levels of subjective behavioural and cognitive engagement with the SIV-ChatGPT was relatively high, while the affective engagement was moderate.
Conclusions:
The SIV-ChatGPT was feasible and acceptable and demonstrated preliminary effectiveness in increasing SIV uptake among people aged 65 years or above. This study also provided implications to improve the performance of the SIV-ChatGPT. A full-powered randomized controlled trial should be considered to evaluate its efficacy. Clinical Trial: ClinicalTrials.gov NCT06679647
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.