JMIR Publications

ABSTRACT

Background:

Background:

Social media has become a critical channel for public health communication during the COVID-19 pandemic, enabling rapid dissemination of information at scale. However, evi-dence on the effectiveness of digital campaigns in shaping vaccine-related perceptions in low- and middle-income countries remains limited. Mexico, with over 88.6 million active internet users, provides a unique setting to evaluate the impact of targeted campaigns on vaccine attitudes. Ob-jective: To assess the effectiveness of five social media advertising campaigns conducted by the Asociación Mexicana de Vacunología (@Vacunologia) on Facebook and Instagram between 2021 and 2022, focusing on ad recall and perceptions of vaccine importance, safety, efficacy, and con-cerns about side effects.

Methods:

We conducted a retrospective ecological analysis using anony-mized, aggregate-level data from five Brand Lift Studies (BLS) provided by Meta. Campaigns reached 84.9 million accounts and generated 179.4 million impressions with a total investment of USD 215,600. Outcomes were measured as incremental “lift” in recall and perceptions between exposed (test) and unexposed (control) groups. Statistical analyses included one-way ANOVA, unpaired t-tests, and Fisher’s Exact Tests. Results are reported as mean differences with 95% con-fidence intervals (CI). Statistical significance was defined as ≥2 percentage points lift with ≥90% probability.

Results:

Across campaigns, ad recall increased significantly in four of five campaigns (mean lift 4.0 percentage points, 95% CI: 2.1–5.9, p<0.001). The women-focused campaign (ages 18–44, major cities) achieved the highest recall (11.6 points, 95% CI: 9.8–13.4). Perceived im-portance of vaccination improved in three of four campaigns where measured (mean lift 2.4 points, 95% CI: 1.1–3.7). Safety perceptions showed modest but positive gains in two campaigns (mean lift 1.9 points, 95% CI: 0.2–3.6), while efficacy perceptions improved only marginally (0.9–1.1 points, 95% CI: –0.5–2.7). Concerns about side effects did not significantly change (–0.6 points, 95% CI: –1.8–0.6). Subgroup analysis indicated younger audiences (18–34 years) exhibited stronger positive shifts in recall and importance compared to older groups (p=0.005 for efficacy by age). No significant differences were observed by sex.

Conclusions:

Social media advertising campaigns in Mexico effectively increased ad recall and improved perceptions of vaccine im-portance and safety, particularly among younger audiences and women in urban areas. However, changes in efficacy perceptions and concerns about side effects were limited. Importantly, no di-rect behavioral outcomes (e.g., vaccination uptake) were measured; the findings reflect attitudinal shifts rather than confirmed vaccination behavior. The innovation and implications of this study lie in evaluating large-scale, real-world digital vaccine campaigns in Latin America using ex-perimental BLS data. Findings highlight that audience segmentation (e.g., women 18–44) and younger demographics yield stronger perceptual shifts, suggesting that tailored digital strategies can complement traditional public health communication. While perceptual changes are not equivalent to behavior, they represent critical precursors to vaccine uptake. Future campaigns should integrate digital analytics with epidemiological data to assess real-world health out-comes.

Objective:

Therefore, the objective of this study is to evaluate the effectiveness of five social media advertising campaigns on COVID‑19 vaccination conducted in Mexico through Facebook and Instagram. Specifically, we aim to assess their impact on ad recall, per-ceived importance, safety, efficacy, and concerns about side effects, and to determine which audiences (by age and sex) were most responsive. Our working hypothesis is that targeted campaigns, particularly those directed at younger populations and women, would generate stronger positive shifts in vaccine-related perceptions compared to general population campaigns.

Methods:

A retrospective and ecological study was conducted using secondary data from Brand Lift Studies (BLS) provided by Meta for campaigns run in 2021 and 2022. All BLS results are aggregated and anonymized. BLS results for Meta are disaggregated by age and gender. Each audience member is likely to be presented with only one question in their newsfeed. This implies that all creative content should align with the BLS questions, as there is no control over which creative or question they will encounter. Facebook con-siders any Brand Lift result that shows a 2-point or higher increase with at least a 90% probability of a brand lift to be statistically significant. Below are several questions to consider if a brand lift result greater than 2 points is observed. For the implementation of the BLS programs, an agreement was signed for the “Scaled Program Support” Program Participation between the Facebook, Inc and Aso-ciación Mexicana de Vacunología. The results provided by Facebook do not contain personal information about the users, they are only statistics about the BLS as presented in tables 1, 2 and 3, which were used to obtain the statistical results. The data was obtained by the BLS tools of Facebook Inc. in accordance with its privacy policies [11] The Asociación Mexicana de Vacunología AC accepted and approved the use of the results of the BLS performed for the realization of this study. The Mexican Association of Vaccinology (@Vacunologia) participated in Facebook’s 2021 and 2022 Scaled Workshop SBCC Program. As part of this program, the AMV re-ceived Facebook Ad Credits and support to conduct a public health campaign on Meta. The campaign characteristics are detailed in Table 1. Table 1. Characteristics of five Brand Lift Studies (BLS) to promote COVID‑19 vaccination in Mexico, 2021–2022. The table summarizes problem statements, behavior change and communi-cation goals, target audiences, survey questions, advertising objectives, campaign duration, and investment. We analyzed information from five Brand Lift Studies (BLS) conducted by the Mexican Association of Vaccinology on Facebook and Instagram (@Vacunologia). BLS are experimental tools developed by Meta to evaluate the impact of advertising campaigns independently of other marketing activities. They work by randomly dividing a rep-resentative sample of eligible users into two groups: a test group, which is exposed to the ads, and a control group, which is not. Both groups are then asked standardized poll questions, and the difference in responses indicates the incremental effect, or “lift,” at-tributable to the campaign [5,6]. Randomization and audience segmentation are performed automatically by Meta’s platform, using predefined targeting criteria such as age, sex, location, or interests. This design ensures internal validity and allows causal inference about the effect of ad ex-posure [7]. Lift is reported as the percentage-point difference between test and control groups, accompanied by a confidence estimate. Outcomes typically include awareness, recall, or perceptions related to the advertised content. The study population consisted of adult Facebook and Instagram users in Mexico who were eligible to be exposed to the campaigns according to predefined targeting criteria (e.g., all adults nationwide, or women aged 18–44 in major cities). Randomization was performed automatically by Meta’s Brand Lift Study (BLS) platform at the user ID level, assigning individuals to either the test group (with the opportunity to see the ads) or the control group (withheld from exposure). This design ensured balance between groups and minimized systematic differences at baseline. The only demographic variables available for analysis were age group and sex, which were provided in disaggregated form in the BLS reports. Because the data were anonymized and aggregated, no addi-tional covariates were accessible. There are different types of lift tests: Conversion Lift (sales, leads, engagement), Brand Lift (awareness and attitudes), and Experiments (custom questions on brand or campaign objectives) [8]. In our case, the BLS measured outcomes relevant to vaccination, such as ad recall, perceived importance, safety, efficacy, and concerns about side effects. These results—whether positive or negative—provide valuable insights into the effec-tiveness of digital communication strategies for vaccines. In this study, the conversion rate was defined as the proportion of respondents in each group who provided a positive response to a Brand Lift Study (BLS) survey item. The expected conversion rate was estimated from the control group’s responses, representing the baseline that would have been observed in the test group without ad exposure. Lift was calculated as the difference between the observed conversion rate in the test group and the expected rate from the control group, with ≥2 percentage points and ≥90% probability considered statistically significant. Lift is widely used because it offers a standardized way to compare campaigns across contexts. Formally, it represents the relative change in outcomes among exposed users compared to the expected rate had they not been exposed. However, when baseline rates are very low, even small absolute differences can produce large lift values, which may overstate effectiveness. Despite this limitation, lift remains an informative indicator of advertising performance, particularly in digital public health interventions where direct behavioral outcomes (e.g., vaccination uptake) are difficult to measure [9]. The Brand Lift Study survey included two types of questions depending on the construct being evaluated. The recall question offered three response options: Yes, No, and I'm not sure. In contrast, questions assessing perceived importance, safety, side ef-fects, and efficacy were based on five-point Likert-type scales. For example, the safety perception item included the options: Very safe, Somewhat safe, Slightly safe, Not safe, and I don't know. Depending on the specific construct, the term “safe” was replaced with “important,” “effective,” or the corresponding attribute. Changes in responses were categorized as either positive or negative shifts in perception, depending on whether the response moved toward or away from more favorable attitudes (e.g., from “Somewhat safe” to “Very safe” versus from “Very safe” to “Not safe”). These directional changes were interpreted as proximal indicators of positive or negative behavior change, con-sistent with established frameworks in behavioral science. The Meta reports utilize the Basic Bayesian Lift methodology, which provides es-timates of confidence rather than p-values. The Basic Bayesian results closely resemble frequentist results but offer confidence estimates. The observed difference between control and test groups, not attributed to chance but instead attributed to a greater than 90% chance of brand lift being observed, is considered to have high confidence. If the ad were shown to a random sample of the target audience, there would be a 90% likelihood of observing the same lift from the control to the test group. Several factors are taken into consideration when determining the chance of brand lift being observed, including the number of respondents answering the question (test group sample size), control response rate (control group sample size, often referred to as 'base'), and the size of the lift (difference between control and test group scores). As the sample size increases for both groups, the scores become more stable. With more stable scores, smaller differences can be confidently declared as statistically significant. The Bayesian statistical method is generally equivalent to frequentist statistical methods in most cases but offers a more conservative measure of confidence when there are known sources of bias or systematic errors. Therefore, observing any statistically significant positive shift (a "lift" of 2% points or more) in vaccine attitudes as a result of an online campaign alone is a significant achievement and worthy of celebration. In this study we present the results provided by Meta company. The outcomes measured were: (1) ad recall, (2) perceived importance of vaccination, (3) perceived safety, (4) perceived efficacy, and (5) concerns about side effects. These were assessed through Brand Lift Study poll questions. No direct behavioral outcomes (e.g., actual vaccination uptake) were available. Because Brand Lift Studies randomly assign users to test and control groups and each respondent is shown only one survey item, the number of re-sponses per subgroup (age × sex × question) is sometimes small. These small n’s reflect the survey design and response distribution, not missing data. These subgroup sizes are inherent to the BLS survey design, where each respondent answers only one question, and therefore should be interpreted as design-driven rather than as missing or incomplete data Statistical analyses included one-way ANOVA to compare mean scores across campaigns, unpaired t-tests for sex differences, and Fisher’s Exact Tests to examine as-sociations between campaign characteristics and the probability of achieving a brand lift (≥80% or ≥90% confidence). Ethical Considerations. Ethics review and approvals: This study was based exclu-sively on secondary, anonymized, and aggregate data provided by Meta’s Brand Lift Studies platform. According to national and international ethical guidelines (e.g., the Declaration of Helsinki and CIOMS International Guidelines), research using anony-mized secondary data that does not involve direct human participation is exempt from formal ethics committee review. The Asociación Mexicana de Vacunología, as the coor-dinating institution, authorized the use of these data for academic and research purposes. Informed consent: Because no individual-level data were accessed and no direct interaction with participants occurred, informed consent was not required. The original data collection by Meta was conducted under its own user agreements and privacy pol-icies, which include provisions for the use of anonymized, aggregate data for research purposes. Privacy and confidentiality: All data analyzed were anonymized and ag-gregated prior to researcher access. No personally identifiable information (PII) was available to the research team. Data were handled in accordance with Meta’s privacy policies and the Asociación Mexicana de Vacunología’s internal data protection standards. Compensation: No participants were recruited directly for this study, and therefore no compensation was provided. Participant identification in images: No images or sup-plementary materials containing identifiable individuals are included in this manuscript.

Results:

Table 2 summarizes the results of the reports from the 5 Brand Lift Studies (BLS) provided by Meta company. Table 2. Results of five Brand Lift Studies (BLS) evaluating COVID‑19 vaccination campaigns on Facebook and Instagram in Mexico, 2021–2022. Metrics include campaign objectives, duration, investment, reach, frequency, impressions, cost per thousand impressions (CPM), and outcomes on ad recall, perceived importance, safety, side effects, and efficacy. Table 3 displays the results of the campaigns from the 5 Brand Lift Studies by sex and age group. It also indicates those that are likely to result in a change in behavior due to the campaign. Where we observe in black that the possibility of the increase being caused by the brand lift is 90% or greater and in gray 80% or greater. Table 3. Results of five Brand Lift Studies (BLS) on COVID‑19 vaccination campaigns in Mexico, 2021–2022, stratified by sex and age group. Presents incremental lift in ad recall, perceived im-portance, safety, side effects, and efficacy, highlighting subgroups most likely to show behavior change. Statistically significant positive shift a "lift" of 2% points or more. The comparison of scores by BLS was carried out, the mean and standard deviation are presented in the following table, a statistically significant difference was found in the mean of the ad recall (p<0.001) and in the mean of the safety score (p=0.049) between the campaigns. The comparison of scores by BLS was conducted, and the mean and standard deviation are presented in table 4. A statistically significant difference was found in the mean of the ad recall (p < 0.001) and in the mean of the safety score (p = 0.049) between the campaigns, as shown in figure 1 and figure 2. Table 4. Comparison of mean scores (with standard deviations) across five Brand Lift Studies (BLS) of COVID‑19 vaccination campaigns in Mexico, 2021–2022. Outcomes include ad recall, safety, efficacy, importance, and side effects. Statistical significance was assessed using one‑way ANOVA. Figure 1. Incremental lift in standard ad recall across five Brand Lift Studies (BLS) of COVID‑19 vaccination campaigns conducted on Facebook and Instagram in Mexico, 2021–2022. Figure 2. Incremental lift in perceived safety of COVID‑19 vaccines across five Brand Lift Studies (BLS) in Mexico, 2021–2022. The comparison by age group was conducted, and the mean and standard deviation are presented in table 5. A statistically significant difference was found in the mean ef-ficacy score (p = 0.005) between the age groups, as shown in figure 3. Table 5. Comparison of Brand Lift Study (BLS) results by age group for COVID‑19 vaccination campaigns in Mexico, 2021–2022. Outcomes include ad recall, safety, efficacy, importance, and side effects. Statistical significance was assessed using ANOVA. Figure 3. Incremental lift in perceived efficacy of COVID‑19 vaccines by age group across five Brand Lift Studies (BLS) in Mexico, 2021–2022. The comparison by sex was conducted for each campaign, and the mean and standard deviation are presented in table 6. No statistically significant differences were found in the scores of any of the items between the sex groups (p > 0.050). Table 6. Comparison of Brand Lift Study (BLS) results by sex for COVID‑19 vaccination cam-paigns in Mexico, 2021–2022. Outcomes include ad recall, safety, efficacy, importance, and side effects. Statistical significance was assessed using unpaired t‑tests. An evaluation was conducted to assess the association between the probability of behavior change by the campaign. A statistically significant association was found be-tween the probability of standard ad recall of 90% or more (p = 0.006) and 80% or more (p = 0.032) with the campaign, all the evalutions are presented in table 7.   Table 7. Association between probability of behavior change (≥80% or ≥90% chance of brand lift) and campaign characteristics in five Brand Lift Studies (BLS) of COVID‑19 vaccination cam-paigns in Mexico, 2021–2022. Outcomes include ad recall, importance, safety, efficacy, and side effects. Statistical significance was assessed using Fisher’s Exact Test. An evaluation was conducted to assess the association between the probability and age group. No statistically significant association was found between the probabilities of the items and the age groups (p > 0.050), all the evaluations are presented in table 8. Table 8. Association between probability of behavior change (≥80% or ≥90% chance of brand lift) and age group in five Brand Lift Studies (BLS) of COVID‑19 vaccination campaigns in Mexico, 2021–2022. Statistical significance was assessed using Fisher’s Exact Test. An evaluation was conducted to assess the association between the probability and sex. No statistically significant association was found between the probabilities of the items and sex (p > 0.050), all the evaluations are presented in table 8. Table 9. Association between probability of behavior change (≥80% or ≥90% chance of brand lift) and sex in five Brand Lift Studies (BLS) of COVID‑19 vaccination campaigns in Mexico, 2021–2022. Statistical significance was assessed using Fisher’s Exact Test.

Conclusions:

This study provides novel evidence on the effectiveness of large-scale social media vaccine campaigns in a Latin American context. By analyzing five campaigns conducted in Mexico, we demonstrate that digital advertising can significantly increase ad recall and improve perceptions of vaccine importance and safety, particularly among younger audiences and women in urban areas. These findings highlight the value of audience segmentation and message targeting in enhancing the impact of digital health commu-nication. While perceptual changes are not equivalent to vaccination behavior, they represent critical precursors to uptake and demonstrate the potential of digital platforms to complement traditional public health strategies. This contribution is especially relevant for low- and middle-income countries, where evidence on digital health interventions remains limited. Future research should integrate digital analytics with epidemiological data to assess whether these perceptual shifts translate into real-world vaccination out-comes. In this study, we emphasize the importance of continuing studies to evaluate and understand how digital media influence health and how these tools can be effectively utilized for behavioral change in health, as discussed in existing literature. Additionally, it is crucial to identify communication differences by conducting studies in low- and middle-income countries, across different geographic locations, and targeting specific audiences. While our campaigns generated impacts on users and elicited interest in vaccination advertisements, we acknowledge the limitation of not being able to measure the final outcome, particularly the variation in vaccination rates in the country and whether the campaigns contributed to its increase or decrease. This challenge was compounded by the simultaneous implementation of offline and online campaigns by other organizations. The results of our study indicate that while vaccination advertisements garnered interest and messages about COVID-19 vaccination were remembered, we did not detect a significant change in behavior through the BLS tool of meta/facebook. Nevertheless, our experience managing the @vacunologia networks revealed that the advertisements sparked comments and controversy among social media users. Through statistical analysis, we conclude that campaigns designed for specific audiences, such as the one targeting women aged 18-44 years in the most populated cities in the country, generate a greater impact. This strategic decision was made with the aim of reaching mothers, who typically play a significant role in managing family health in Mexico. When observing the data and statistical tests by age, it can be inferred that the younger the age, the greater the probability of generating a behavioral change, which was not observed by sex. Additionally, the differences observed in the results of the first campaign, including investment, costs, and outcomes, were much better than the other four campaigns, which had between 2 and 4 times higher costs. We do not have a clear explanation for this discrepancy, whether it was due to differences in design, objectives, investment, artistic elements, or campaign messaging. We believe that these differences may be attributed to various factors such as the stage of the pandemic, implemented control measures, social media usage patterns, and the initiation of vaccination efforts. There is no doubt that there is still much work to be done, but it is of great importance for health specialists to continue collaborating and engaging audiences in digital media. By creating experts in the dissemination of medical information and generating be-havioral changes, we can improve the health of the population. However, to identify the best strategies, it is essential to generate evidence to use as a basis for creating, designing, and evaluating successful campaigns that lead to real behavioral changes in health worldwide.