Behavioural interventions to reduce vaccine hesitancy driven by misinformation on social media

Correspondence to: K Ruggeri [email protected]

Key messages

• Substantial evidence shows the negative effects of vaccine misinformation on social media

• Evidence on the effectiveness of interventions to correct or mitigate misinformation is considerably more limited and rarely includes measures of true vaccine uptake

• The evidence available does indicate ways forward to develop better methods, particularly those that would be less likely to backfire in the way that blanket social media bans have

• The need for such actions is urgent

Vaccine misinformation on social media has strong effects on behaviour, and the evidence base for interventions to reduce these effects is limited, but better approaches to evidence generation are possible, say Kai Ruggeri and colleagues

Effective population level vaccination campaigns are fundamental to public health.¹²³ Counter campaigns, which are as old as the first vaccines,⁴ disrupt uptake and can threaten public health globally.⁴ In 2019, public health researchers linked increases in measles cases with the proliferation of global anti-vaccine campaigns.⁵ Some of these campaigns originated offline but were later amplified and expedited through social media, resulting in real world harms.⁶ Though crises and genuine safety concerns can also lower vaccine uptake,⁷⁸ the return of measles after aggressive anti-vaccine campaigns prompted the World Health Organisation to list vaccine hesitancy among the greatest threats to global health (box 1).¹⁴

Box 1

Vaccine hesitancy

Here, we use the term “vaccine hesitancy” as originally defined by WHO⁹: a “delay in acceptance or refusal of vaccination despite availability of vaccination services.” This definition, plus the expanded description of variation based on time, place, and population, gives a broad understanding but also allows us to distinguish vaccination behaviour from the underpinning psychological, environmental, and structural aspects influencing behaviour. This definition is most commonly used in literature pre-dating covid-19. Though there are debates about the most appropriate terms (“vaccine demand” is an alternative, for example) or updated definitions (including the updated WHO version) to use, applying the original WHO definition ensured that we captured relevant insights and evidence from the literature.^10111213 We also identify disinformation as a form of misinformation that deliberately seeks to mislead or otherwise disrupt understanding, although we cover both broadly under misinformation for the purposes of this paper.

Anti-vaccine campaigns proliferated during the covid-19 pandemic¹⁵ with undeniable effects including substantial increases in covid related illness and death.¹⁵¹⁶¹⁷ Even before March 2020, vaccine hesitancy was directly linked to misinformation (false, inaccurate information promoted as factual) spread on social media.¹⁸ Once covid-19 reached pandemic status, social media was acknowledged as the epicentre of misinformation leading to hesitancy,¹⁹²⁰ and consequently, interventions to tackle hesitancy have globally focused on delivery through social media.²¹²²

Despite unprecedented levels of vaccine access and nearly real time communication on the development and availability of vaccines in 2020-21, public health officials struggled to keep pace with misleading or inaccurate content online.²³ As guidelines shifted with the emergence of new information, policy decisions were often perceived by individuals and groups who are prone to distrust or refute government messaging as a response not to evidence but to mistakes or lack of expertise.²⁴²⁵ A 2021 randomised controlled trial (RCT) found that exposure to misinformation about covid-19 vaccines lowered the intent of recipients to vaccinate, even among those reporting before exposure that they would “definitely” accept vaccination.²¹ Similar findings globally indicate that social media dynamics exacerbated the sharing of misinformation, reduced vaccination rates, undermined trust in reliable information, magnified polarisation, and damaged the perceived credibility of institutions.^2627282930 These challenges remain today.

Approaches to reducing vaccine hesitancy

Standard behavioural approaches to encourage vaccination include mandatory vaccination and regulation for healthcare professionals, incentives, public health communication campaigns, and engaging trusted leaders.³¹ Contemporary methods have started to be implemented on social media,^10313233 including debunking (fact checking specific claims after they have reached social media users) and “pre-bunking,” a behavioural approach in which users are taught about how “fake news” works before exposure.³⁴ Other intervention types include warning (“inoculating”) people about manipulation tactics using non-harmful exposure as a tool to identify misinformation, and using accuracy prompts to trigger people to consider the truthfulness of material they are about to share on social media platforms, without stopping them from posting.³⁵ Recognising the intense effect of social media on vaccine hesitancy,³⁶ the Africa Centres for Disease Control and Prevention developed a toolkit to assist countries in social media strategies aimed at encouraging vaccination.³⁷

Such behavioural approaches to misinformation on social media have shown promise in reducing the sharing of disinformation and misinformation (box 1) and in changing people’s beliefs,³⁸ but less clear is their effect on vaccination uptake. Investigating the success of such interventions on uptake is essential because there is an established link between social media exposure and offline beliefs that vaccines are unsafe.³⁹ It is clear, however, that providing fact based probabilistic information alone fails to meaningfully increase uptake⁴⁰ and might even backfire.⁴¹ Factors such as low trust in governments and health institutions are likely to be instrumental in derailing effective immunisation programmmes.⁴² Multiple drivers and barriers to vaccine uptake must therefore be considered when developing effective tools.

Evaluating interventions to reduce vaccine hesitancy

We identified 30 studies evaluating interventions to tackle misinformation on social media that explicitly captured real world behavioural outcomes (see supplementary information on bmj.com). After excluding 19 studies for lack of external validation, only 11 published interventions were left (table 1; box 2; fig 1). There is clearly insufficient evidence from field studies on social media interventions. Only two interventions that met all criteria came from outside the US (Israel⁴³ and Nigeria⁵³), with a recent study conducted in Nigeria producing perhaps the most robust level of evidence.

Table 1

Interventions to reduce vaccine hesitancy through social media that assessed real world vaccine behaviours

Study and countryVaccineDemographics    Type of hesitancyMethodsIntervention and outcome

Chodick et al. (2021)43 Israel	HPV	Mothers of teen daughters	Lack of knowledge	RCT	Facebook adverts addressing issues and concerns about HPV and the vaccine with inoculation elements had positive effects on vaccination rates in the second quarter by socioeconomic status, a negative effect in the lowest quarter, and no effect in higher socioeconomic levels
Brandt et al. (2020)44 USA	HPV	College students	Underestimating severity and susceptibility	Questionnaires	Facebook group posts, weekly emails, and open- ended reflection appealing to personal benefits and health education showed increased knowledge but had no significant effect on self-reported vaccination
Bonnevie et al. (2020)45 USA	Flu	African American and Hispanic US residents	Group norms	Before and after	Influencer created content campaigns showed increases in positive vaccine beliefs and small but non-significant effects on self-reported vaccination
O’Leary et al. (2019)46 USA	Tdap, flu	Pregnant women	Concerns about vaccines due to providers’ lack of time and resources to inform	RCT	A website with vaccine information and interactive social media components had medium positive effects on vaccination rates specifically among pregnant women
Ortiz et al. (2019)47 USA	HPV	Adolescents	Lack of knowledge	Before and after	Health facts appearing in Facebook news feed every 4 to 5 days showed increased knowledge but no effect on vaccinations
Mohanty et al. (2018)48 USA	HPV	Adolescents	Lack of knowledge and awareness	Intervention without control	Facebook advertisement campaigns directing to a website to schedule vaccination reached a large audience but had no effects on vaccination rates.
Sundstrom et al. (2018)49 USA	HPV	College-age women	Underestimating severity and susceptibility	Focus groups and survey	Information posts on several social media outlets increased self-reported vaccination, though no statistical test was conducted
Huang et al. (2018)50 USA	Flu	Undergraduate students		Difference in difference	Social media campaign appealing to students’ community identity had positive effects on vaccination rates
Glanz et al. (2017) 51 USA	Multiple early childhood vaccines	Pregnant women	Misinformation on social media	RCT	Website with vaccine information and interactive social media components reduced the days undervaccinated of infants with medium effects specifically among pregnant women
Shropshire et al. (2013)52 USA	Flu	University students	Concerns about vaccines, underestimating susceptibility	Post-vaccination survey	Multimedia campaign about flu vaccine including social media outlets had positive effects on vaccination rates
Evans et al. (2023)53 Nigeria	COVID-19	Facebook users	Lack of positive social norms	Quasi-experiment	Social media campaign focused on promoting pro-vaccination social norms using social influencers had positive effects on self-reported vaccination.

Box 2

A robust research agenda for social media interventions is needed

Studies on reducing vaccine hesitancy can inform broader vaccination messaging campaigns. We recommend some critical features that should be included for all interventions (fig 1). Details about the type of vaccine and the target group should be highly specific, as should the form of hesitancy and misinformation,⁵⁴ the misinformation source, and details of the method (especially for RCTs). Real world effects (ideally direct and broader public health indicators) must be captured. These aspects are relevant for better understanding of what might work and for what reasons and for comparison between methods and social media platforms. The most informative approach would cover the first three levels of evidence under the THEARI system (theoretical, empirical, and real world).⁵⁵ Over time, these would accumulate to the fourth (replicated) and fifth (well established impact) levels, maximising the potential for predictive validity of applications to public policy (for example, greater confidence in anticipated effects). Doing so would exceed the criteria listed in a recent call for a gold standard for trials tackling vaccine hesitancy.⁵⁶

Fig 1. Critical aspects of studies that are necessary to better understand and compare possible interventions that tackle vaccine hesitancy on social media (described in box 2). The examples of what specifically to measure and observe are non-exhaustive. All aspects should be included for maximum value, though behavioural and population health outcomes should be prioritised. Boxes with thick outlines and darker shading are considered absolutely necessary, though all boxes are strongly recommended.

Some evidence exists from field studies that did not directly target misinformation but provided accurate and useful information about vaccines. These were typically posted on interactive websites or directly in social media feeds. Changes in attitudes, knowledge, and social media engagement were reported for several information campaigns using social media advertisements in before and after designs and designs without control groups.^44454748 But these studies showed no changes in vaccination behaviours. Only two studies, both RCTs, showed that tackling vaccination concerns and providing information in interactive formats on a website increased vaccination⁴⁶ and reduced days undervaccinated.⁵¹

Two interventions that increased vaccination uptake used targeted messages through Facebook adverts⁵⁷ and personalised influencer content.⁵⁸ But each had limitations. In the first, vaccination rates increased only among families of medium-low socioeconomic status, indicating narrow effectiveness. In the second, the measures were primarily attitudinal, and behavioural outcomes were self-reported.

Linking online campaigns to true behaviours in field studies is a major challenge. This is partly because running well controlled studies is simpler in laboratory experiments and surveys, and because capturing distal effects (true behaviours) of web based studies is often impossible. But showing some measures of success in increasing uptake is necessary to justify investment to develop interventions at scale. Therefore, it is critical that researchers of vaccine behaviours and associated campaigns engage directly with clinics and public health agencies to improve the ecological and external validity (that is, real world, observed outcomes) of interventions (box 2).

Developing better interventions to confront vaccine hesitancy on social media

Few interventions among those we evaluated captured evidence about real world behaviours, so there is currently no gold standard toolkit that public health agencies can refer to.⁵⁶ But the evidence available can inform future tool development. We provide 10 insights based on the existing evidence that should help provide a clearer and more specific, evidence driven toolkit of approaches to reduce vaccine hesitancy.

• Negative sentiments on social media might increase vaccine hesitancy faster than interventions reduce it³⁹⁵⁹—Although vaccine hesitancy can stem from many sources, including mass media and political rhetoric as well as genuine safety concerns, there is ample evidence of a proliferation of anti-vaccine messages on social media leading to organised offline actions and increased hesitancy.⁶⁰ This is driven by several social and individual factors, as well as foreign disinformation campaigns and bots.⁶¹⁶² There is less evidence that efforts to specifically mitigate misinformation have had a reliable effect on real world vaccine uptake.

• Messaging seems to work best when it is tailored to what groups know and care about—Two relatively large vaccination interventions, one for flu, one for human papillomavirus (HPV), found modest effects through targeting specific groups. Broad campaigns through all forms of media tend to be generally effective. But once misinformation and conflicting views are prevalent, speaking directly to audiences, knowing the reasons for hesitancy, and framing information in a way that matters to individuals are critical.⁵⁴⁶³⁶⁴

• Simple messaging about benefits and risk based on probabilities is not enough—Filling vaccine knowledge gaps has direct benefits, such as supporting informed decisions,⁶³ but messaging with information about benefits, harms, and associated probabilities is insufficient to resolve vaccine hesitancy.⁴⁰ Some possible explanations include lack of trust and cultural values, which have a major effect on how scientific data are interpreted and accepted through social media.⁶⁵ Messages must be conveyed in a way that affirms individual cultural values,⁶⁶ dealing with topics of importance to individuals, not only health facts, using credible sources of information.⁶⁷ Visual imagery also helps deliver effective messages,⁶⁸ and other components should be considered, such as health literacy, simplified language, and context specific features like age or language.

• Correct misinformation to both parents and their children—Parents, especially mothers, play a major part in child vaccination.⁵⁸ One large campaign carried out on Facebook directed at mothers of teens had some positive effects in specific income groups, but the overall effect was minimal to null. Addressing parents is clearly of value, but young people also seek out information online for themselves.⁶⁹ Thus, the information teens are presented with online likely needs similar but unique safeguards against misinformation to the ones discussed. Directly involving parents and young people in the design of messaging might strengthen the effectiveness of these campaigns.⁷⁰

• Trust matters: the message,⁷¹the messenger,⁶³and the (vaccinated) provider⁵⁷—Substantial amounts of anti-vaccine messaging comes from non-medically qualified, or non-expert, voices on social media.⁷² Thus, medical professionals are under-represented when it comes to legitimate and accurate information about vaccines on social media,⁷³ despite typically being the most trusted sources (especially among parents⁷⁴). Trust is potentially the most distinct characteristic of successful vaccination campaigns,⁵⁰⁷⁵ and this includes ones delivered on social media.⁷⁶ Ultimately, the source of the message—whether a healthcare provider, a politician, or a social media influencer—is likely to have a major role in whether individuals and communities deem information to be credible.⁵⁷⁶³ These interventions have great potential when delivered to the right population.³⁰ The US state of New York, for example, was a stronghold of anti-vaccine sentiment before 2020, which pivoted after the arrival of covid-19 vaccines. Despite intensifying hesitancy and misinformation waves, the state outpaced the national average for vaccination rates, benefiting both public health⁷⁷ and economic returns.⁷⁸ This success is attributed to strategies that, on top of tackling misinformation, also cultivated trust in the source of the vaccine, the messenger, and the provider, involving use of the military (a highly trusted organisation in the US), diverse community messengers (including community health workers and faith leaders), and a broad network of vaccination locations.^79808182 These aspects must be ensured in public campaigns aimed at specific groups, especially those who are vaccine hesitant.⁶⁴

• Debunking efforts have shown mixed effects on social media—Distributing information from public institutions or providing objective information from third parties might help to counter misinformation, reduce the intention to spread misinformation, and promote health behaviours, but the process is not always smooth. The backfire effect is a concerning pattern in which disproving misinformation reinforces it and deepens false beliefs.⁸³ But this phenomenon is not consistently observed in practice. Effects of debunking might be highly dependent on the recipient’s background knowledge and beliefs, methods of presentation, and what other information is viewed.⁸⁴⁸⁵ For now, the immediate priority for this approach is to determine if inoculation based on pre-bunking can reliably reduce the noxious effects of next widespread waves of misinformation and resulting misbeliefs.

• Raising the quality and visibility of reliable health information can counter misinformation—Providing simple probabilistic information is not likely to completely counteract misinformation on its own, but providing information on how, where, and when to get a vaccine does.⁷⁶ The high volume of misinformation appearing in online searches, however, can override more reliable sources,⁸⁶ limiting the effectiveness of high-quality information campaigns.⁸⁷ To increase campaign effectiveness, interactive designs and visual aids such as posters or videos help target populations see and engage with accurate, accessible information throughout a campaign.⁵¹ Making it easier to find such materials is paramount, starting with search engine optimisation to increase visibility of campaigns.

• Framing of vaccine messages matters—How public campaign messaging is framed affects health decision making. A public campaign can’t cover all vaccines, diseases, populations, and reasons for hesitancy; framing messages to be directly relevant to a populations’ needs (benefits and risks specific to their group) have resulted in significant increases in uptake.⁸⁸ Positive and negative framing effects are not equivalent,⁸⁹ however, and this should be factored into messaging decisions.

• Blanket bans can drive groups and activities underground—Broad social media bans of individuals or of specific content can paradoxically result in the spread of misinformation and can galvanise problematic echo chambers by driving discussion into private social media groups or closed forums.⁹⁰⁹¹ Such closed environments are unlikely to include different viewpoints or corrective information, so misinformation is more likely to be reinforced. Rather than rely on outright bans, policy makers and content managers should explore methods that limit the spread and influence of misinformation.⁹²

• Social media platforms need to be part of the solution—If social media platforms are the epicentre of misinformation, then social media companies need to be part of the solution.³⁹ During the covid-19 pandemic, social media platforms took a more interventionist approach to content moderation than before (and, in some cases, removed or limited covid-19 misinformation and conspiracies).⁹³ Some of these approaches are now being rolled back, and social media researchers’ access to data about behaviour on the global platform X (formerly Twitter) is being limited. Content labelling and corrective actions have produced some positive effects,⁹⁴ but social media companies should be more proactive in dealing with the proliferation of misinformation on their sites.⁹⁵ We endorse calls³⁹ to make data available and to work with researchers and regulators in all countries to enable developing effective solutions.

It is worth it to get these campaigns right

Misinformation is not new and its noxious consequences are not insurmountable, but its effect on vaccine hesitancy through social media is an urgent global threat to public health. Increasingly robust evidence has shown the drivers and effects of this phenomenon, but few successful, let alone replicated, interventions exist. One important step towards developing more effective interventions is the close monitoring of public perceptions and opinions about vaccination and services. Digital technologies make it possible to analyse large quantities of “social listening data” in real time.⁹⁶ Such information would complement the evidence we already have from a variety of study types, improving the design of new, robust, and appropriately targeted interventions. Because most published interventions focused on attitudes and intentions rather than on actual vaccination, however, there remains an urgent need for direct partnerships between behavioural researchers with healthcare clinics and public health agencies.⁷⁷ Meaningfully developing those partnerships promises direct benefits for more reliable scientific insights that would improve the health and well being of entire populations.

We thank Valentina Cafarelli for assistance with reviewing interventions identified during the search.

Footnote

Contributors and sources: KR conceived the idea. BV and FS were involved in the conceptualisation and delivery of the literature search, with support from YAA. KR and FS reviewed and extracted information from studies. YY, RH, SV, YAA contributed to the first draft of the manuscript. All co-authors reviewed and provided insightful comments to subsequent manuscript drafts. KR and PSB provided earlier drafts of the figure in box 2. All authors approved the final report. KR is responsible for the overall content as the guarantor of this work.

Provenance and peer review: Commissioned; externally peer reviewed.

Patient and public involvement: No patients or general members of the public were asked to participate given this Analysis did not relate to clinical care or health services. The focus of the article was to summarise available evidence and provide directions for future research.

Competing interests: All authors confirm that they have no conflicts or competing interests in contributing to this manuscript.

The article is part of a collection that was proposed by the Advancing Health Online Initiative (AHO), a consortium of partners including Meta and MSD, and several non-profit collaborators (https://www.bmj.com/social-media-influencing-vaccination). Research articles were submitted following invitations by The BMJ and associated BMJ journals, after consideration by an internal BMJ committee. Non-research articles were independently commissioned by The BMJ with advice from Sander van der Linden, Alison Buttenheim, Briony Swire-Thompson, and Charles Shey Wiysonge. Peer review, editing, and decisions to publish articles were carried out by the respective BMJ journals. Emma Veitch was the editor for this collection.

¹ Carter A Msemburi W Sim SY. Modeling the impact of vaccination for the immunization Agenda 2030: Deaths averted due to vaccination against 14 pathogens in 194 countries from 2021 to 2030. Vaccine 2023; S0264-410X(23)00854-X. 10.1016/j.vaccine.2023.07.033   37537094

² Centers for Disease Control and Prevention. Why CDC is involved in global immunization. 2023. https://www.cdc.gov/globalhealth/immunization/why/index.html.

³ Sim SY Watts E Constenla D Brenzel L Patenaude BN. Return on investment from immunization against 10 pathogens in 94 low- and middle-income countries, 2011-30. Health Aff (Millwood) 2020; 39: 1343 - 53. 10.1377/hlthaff.2020.00103   32744930

⁴ Mutombo PN Fallah MP Munodawafa D. COVID-19 vaccine hesitancy in Africa: a call to action. Lancet Glob Health 2022; 10: e320 - 1. 10.1016/S2214-109X(21)00563-5   34942117

⁵ Hotez PJ Nuzhath T Colwell B. Combating vaccine hesitancy and other 21st century social determinants in the global fight against measles. Curr Opin Virol 2020; 41: 1 - 7. 10.1016/j.coviro.2020.01.001   32113136

⁶ Dyer O. Measles outbreak in Somali American community follows anti-vaccine talks. BMJ 2017; 357: j2378. 10.1136/bmj.j2378   28512183

⁷ Lam E McCarthy A Brennan M. Vaccine-preventable diseases in humanitarian emergencies among refugee and internally-displaced populations. Hum Vaccin Immunother 2015; 11: 2627 - 36. 10.1080/21645515.2015.1096457   26406333

⁸ Cooper S Gadanya M Kaawa-Mafigir D et al. Using social media to build confidence in vaccines: lessons from community engagement and social science research in Africa. BMJ 2023; 383: e075564

⁹ MacDonald NE SAGE Working Group on Vaccine Hesitancy. Vaccine hesitancy: definition, scope and determinants. Vaccine 2015; 33: 4161 - 4. 10.1016/j.vaccine.2015.04.036   25896383

¹⁰ Larson HJ Gakidou E Murray CJL. The vaccine-hesitant moment. N Engl J Med 2022; 387: 58 - 65. 10.1056/NEJMra2106441   35767527

¹¹ Dudley MZ Privor-Dumm L Dubé È MacDonald NE. Words matter: vaccine hesitancy, vaccine demand, vaccine confidence, herd immunity and mandatory vaccination. Vaccine 2020; 38: 709 - 11. 10.1016/j.vaccine.2019.11.056   31836257

¹² Bussink-Voorend D Hautvast JLA Vandeberg L Visser O Hulscher MEJL. A systematic literature review to clarify the concept of vaccine hesitancy. Nat Hum Behav 2022; 6: 1634 - 48. 10.1038/s41562-022-01431-6   35995837

¹³ World Health Organization. Understanding the behavioural and social drivers of vaccine uptake WHO position paper—May 2022. https://www.who.int/publications-detail-redirect/who-wer9720-209-224.

¹⁴ Nohynek H, Arora N, Binka F, et al. Report of the Expert Advisory Group on SAGE Evaluation (EAGSE). https://cdn.who.int/media/docs/default-source/immunization/sage/sage-evaluation-2019/sage_evaluation_report_2019_print_final.pdf

¹⁵ Unicef. New data indicates declining confidence in childhood vaccines of up to 44 percentage points in some countries during the covid-19 pandemic. https://www.unicef.org/press-releases/sowc_2023_immunization

¹⁶ Karafillakis E Van Damme P Hendrickx G Larson HJ. COVID-19 in Europe: new challenges for addressing vaccine hesitancy. Lancet 2022; 399: 699 - 701. 10.1016/S0140-6736(22)00150-7   35123665

¹⁷ Council of Canadian Academies. Fault lines. The Expert Panel on the Socioeconomic Impacts of Science and Health Misinformation. 26 Jan 2023. https://cca-reports.ca/reports/the-socioeconomic-impacts-of-health-and-science-misinformation/.

¹⁸ Burki T. Vaccine misinformation and social media. Lancet Digit Health 2019; 1: e258 - 9 10.1016/S2589-7500(19)30136-0

¹⁹ Razai MS Chaudhry UAR Doerholt K Bauld L Majeed A. Covid-19 vaccination hesitancy. BMJ 2021; 373: n1138. 10.1136/bmj.n1138   34016653

²⁰ Solís Arce JS Warren SS Meriggi NF. COVID-19 vaccine acceptance and hesitancy in low- and middle-income countries. Nat Med 2021; 27: 1385 - 94. 10.1038/s41591-021-01454-y   34272499

²¹ Loomba S de Figueiredo A Piatek SJ de Graaf K Larson HJ. Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA. Nat Hum Behav 2021; 5: 337 - 48. 10.1038/s41562-021-01056-1   33547453

²² Arechar AA Allen J Berinsky AJ. Understanding and combatting misinformation across 16 countries on six continents. Nat Hum Behav 2023; 7: 1502 - 13. 10.1038/s41562-023-01641-6   37386111

²³ Ruggeri K Stock F Haslam SA. A synthesis of evidence for policy from behavioral science during covid. Nature 2023; https://doi.org/10.1038/s41586-023-06840-9

²⁴ Wegwarth O Hertwig R Giese H Fineberg HV. The impact of nontransparent health communication during the covid-19 pandemic on vaccine- hesitant people’s perception of vaccines. Preprint. 2023 10.31219/osf.io/znwyx.

²⁵ Stephenson-Hunter C Yusuf Y Larson R Campanella J Gutnick DN. What matters to us: bridging research and accurate information through dialogue (BRAID) to build community trust and cultivate vaccine confidence. Prev Med Rep 2023; 34: 102253. 10.1016/j.pmedr.2023.102253   37252070

²⁶ Pierri F Perry BL DeVerna MR. Online misinformation is linked to early COVID-19 vaccination hesitancy and refusal. Sci Rep 2022; 12: 5966. 10.1038/s41598-022-10070-w   35474313

²⁷ Fridman A Gershon R Gneezy A. COVID-19 and vaccine hesitancy: A longitudinal study. PLoS One 2021; 16: e0250123. 10.1371/journal.pone.0250123   33861765

²⁸ Yang K-C. The covid-19 infodemic: Twitter versus Facebook. Big Data Soc 2021; 8: 20539517211013860 10.1177/20539517211013861

²⁹ Wallace J Goldsmith-Pinkham P Schwartz JL. Excess death rates for Republican and Democratic registered voters in Florida and Ohio during the covid-19 pandemic. JAMA Intern Med 2023; 183: 916 - 23. 10.1001/jamainternmed.2023.1154   37486680

³⁰ Larsen BJ Ryan TJ Greene S Hetherington MJ Maxwell R Tadelis S. Counter-stereotypical messaging and partisan cues: Moving the needle on vaccines in a polarized United States. Sci Adv 2023; 9: eadg9434. 10.1126/sciadv.adg9434   37467319

³¹ Cameron-Blake E Tatlow H Andretti B. A panel dataset of covid-19 vaccination policies in 185 countries. Nat Hum Behav 2023; 7: 1402 - 13. 10.1038/s41562-023-01615-8   37414885

³² Pluviano S Watt C Della Sala S. Misinformation lingers in memory: Failure of three pro-vaccination strategies. PLoS One 2017; 12: e0181640. 10.1371/journal.pone.0181640   28749996

³³ Lazer DMJ Baum MA Benkler Y. The science of fake news. Science 2018; 359: 1094 - 6. 10.1126/science.aao2998   29590025

³⁴ Basol M. Towards psychological herd immunity: Cross-cultural evidence for two prebunking interventions against COVID-19 misinformation. Big Data Soc 2021; 8: 20539517211013868 10.1177/20539517211013868

³⁵ Pennycook G Rand DG. Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation. Nat Commun 2022; 13: 2333. 10.1038/s41467-022-30073-5   35484277

³⁶ Osuagwu UL Mashige KP Ovenseri-Ogbomo G. The impact of information sources on COVID-19 vaccine hesitancy and resistance in sub-Saharan Africa. BMC Public Health 2023; 23: 38. 10.1186/s12889-022-14972-2   36609264

³⁷ Africa CDC. Covid-19 social media support kit. https://africacdc.org/download/covid-19-social-media-support-kit/

³⁸ Pennycook G Rand DG. Nudging social media toward accuracy. Ann Am Acad Pol Soc Sci 2022; 700: 152 - 64. 10.1177/00027162221092342   35558818

³⁹ Wilson SL Wiysonge C. Social media and vaccine hesitancy. BMJ Glob Health 2020; 5: e004206. 10.1136/bmjgh-2020-004206   33097547

⁴⁰ Dubé E Laberge C Guay M Bramadat P Roy R Bettinger J. Vaccine hesitancy: an overview. Hum Vaccin Immunother 2013; 9: 1763 - 73. 10.4161/hv.24657   23584253

⁴¹ Nyhan B Reifler J. Does correcting myths about the flu vaccine work? An experimental evaluation of the effects of corrective information. Vaccine 2015; 33: 459 - 64. 10.1016/j.vaccine.2014.11.017   25499651

⁴² Goldenberg MJ. Vaccine hesitancy: public trust, expertise, and the war on science. University of Pittsburgh Press, 2021 10.2307/j.ctv1ghv4s4

⁴³ Chodick G Teper GR Levi S. The impact of a Facebook campaign among mothers on HPV vaccine uptake among their daughters: A randomized field study. Gynecol Oncol 2021; 160: 106 - 11. 10.1016/j.ygyno.2020.10.037   33162176

⁴⁴ Brandt HM Sundstrom B Monroe CM. Evaluating a technology-mediated HPV vaccination awareness intervention: a controlled, quasi-experimental, mixed methods study. Vaccines (Basel) 2020; 8: 749. 10.3390/vaccines8040749   33321975

⁴⁵ Bonnevie E Rosenberg SD Kummeth C Goldbarg J Wartella E Smyser J. Using social media influencers to increase knowledge and positive attitudes toward the flu vaccine. PLoS One 2020; 15: e0240828. 10.1371/journal.pone.0240828   33064738

⁴⁶ O’Leary ST Narwaney KJ Wagner NM Kraus CR Omer SB Glanz JM. Efficacy of a web-based intervention to increase uptake of maternal vaccines: An RCT. Am J Prev Med 2019; 57: e125 - 33. 10.1016/j.amepre.2019.05.018   31471001

⁴⁷ Ortiz RR Shafer A Cates J Coyne-Beasley T. Development and evaluation of a social media health intervention to improve adolescents’ knowledge about and vaccination against the human papillomavirus. Glob Pediatr Health 2018; 5: X18777918. 10.1177/2333794X18777918   29872667

⁴⁸ Mohanty S Leader AE Gibeau E Johnson C. Using Facebook to reach adolescents for human papillomavirus (HPV) vaccination. Vaccine 2018; 36: 5955 - 61. 10.1016/j.vaccine.2018.08.060   30172634

⁴⁹ Sundstrom B Brandt HM Gray L Young Pierce J. It’s my time: applying the health belief model to prevent cervical cancer among college-age women. J Commun Manag (Lond) 2018; 22: 161 - 78 10.1108/JCOM-06-2016-0044

⁵⁰ Huang JJ Francesconi M Cooper MH Covello A Guo M Gharib SD. Community health workers on a college campus: Effects on influenza vaccination. J Am Coll Health 2018; 66: 317 - 23. 10.1080/07448481.2018.1440582   29447623

⁵¹ Glanz JM Wagner NM Narwaney KJ. Web-based social media intervention to increase vaccine acceptance: a randomized controlled trial. Pediatrics 2017; 140: e20171117. 10.1542/peds.2017-1117   29109107

⁵² Shropshire AM Brent-Hotchkiss R Andrews UK. Mass media campaign impacts influenza vaccine obtainment of university students. J Am Coll Health 2013; 61: 435 - 43. 10.1080/07448481.2013.830619   24152021

⁵³ Evans WD Bingenheimer JB Long M. Outcomes of a social media campaign to promote covid-19 vaccination in Nigeria. PLoS One 2023; 18: e0290757. 10.1371/journal.pone.0290757   37713381

⁵⁴ Fasce A Schmid P Holford DL Bates L Gurevych I Lewandowsky S. A taxonomy of anti-vaccination arguments from a systematic literature review and text modelling. Nat Hum Behav 2023; 7: 1462 - 80. 10.1038/s41562-023-01644-3   37460761

⁵⁵ Ruggeri K, van der Linden S, Wang YC, Papa F, Riesch J, Green J. Standards for evidence in policy decision-making. Nature Research Social and Behavioural Science. 5 Jun 2020. go.nature.com/2zdTQIs

⁵⁶ van der Linden S. We need a gold standard for randomised control trials studying misinformation and vaccine hesitancy on social media. BMJ 2023; 381: 1007. 10.1136/bmj.p1007   37146997

⁵⁷ Bartoš V Bauer M Cahlíková J Chytilová J. Communicating doctors’ consensus persistently increases COVID-19 vaccinations. Nature 2022; 606: 542 - 9. 10.1038/s41586-022-04805-y   35650433

⁵⁸ Argyris YA Kim Y Roscizewski A Song W. The mediating role of vaccine hesitancy between maternal engagement with anti- and pro-vaccine social media posts and adolescent HPV-vaccine uptake rates in the US: The perspective of loss aversion in emotion-laden decision circumstances. Soc Sci Med 2021; 282: 114043. 10.1016/j.socscimed.2021.114043   34147269

⁵⁹ Rathje S He JK Roozenbeek J Van Bavel JJ van der Linden S. Social media behavior is associated with vaccine hesitancy. PNAS Nexus 2022; 1: pgac207. 10.1093/pnasnexus/pgac207   36714849

⁶⁰ Martin S Vanderslott S. “Any idea how fast ‘It’s just a mask!’ can turn into ‘It’s just a vaccine!’”: From mask mandates to vaccine mandates during the COVID-19 pandemic. Vaccine 2022; 40: 7488 - 99. 10.1016/j.vaccine.2021.10.031   34823912

⁶¹ Yuan X Schuchard RJ Crooks AT. Examining emergent communities and social bots within the polarized online vaccination debate in Twitter. Soc Media Soc 2019; 5: 2056305119865465 10.1177/2056305119865465

⁶² Walter D Ophir Y. Online foreign propaganda campaigns and vaccine misinformation: a comparative analysis. In: Ginossar T Shah SFA Weiss D, eds. Vaccine Communication Online. Palgrave Macmillan, 2023, https://doi-org.ezproxy.cul.columbia.edu/10.1007/978-3-031-24490-2_6 10.1007/978-3-031-24490-2_6

⁶³ Limaye RJ Holroyd TA Blunt M. Social media strategies to affect vaccine acceptance: a systematic literature review. Expert Rev Vaccines 2021; 20: 959 - 73. 10.1080/14760584.2021.1949292   34192985

⁶⁴ Omer SB Benjamin RM Brewer NT. Promoting COVID-19 vaccine acceptance: recommendations from the Lancet Commission on Vaccine Refusal, Acceptance, and Demand in the USA. Lancet 2021; 398: 2186 - 92. 10.1016/S0140-6736(21)02507-1   34793741

⁶⁵ Kahan D. Fixing the communications failure. Nature 2010; 463: 296 - 7. 10.1038/463296a   20090734

⁶⁶ Ajeigbe O Arage G Besong M Stellenbosch Institute for Advanced Study African Scholars Network. Culturally relevant COVID-19 vaccine acceptance strategies in sub-Saharan Africa. Lancet Glob Health 2022; 10: e1090 - 1. 10.1016/S2214-109X(22)00251-0   35691328

⁶⁷ Richard-Eaglin A McFarland ML. Applying cultural intelligence to improve vaccine hesitancy among black, indigenous, and people of color. Nurs Clin North Am 2022; 57: 421 - 31. 10.1016/j.cnur.2022.04.008   35985729

⁶⁸ Papachrisanthou MM Lorenz RA Loman DG. Increasing immunization adherence among infants of low-income parents: the effects of visually enhanced education. J Nurse Pract 2016; 12: 304 - 10 10.1016/j.nurpra.2015.12.021

⁶⁹ Taba M Ayre J Freeman B McCaffery K Bonner C. COVID-19 messages targeting young people on social media: content analysis of Australian health authority posts. Health Promot Int 2023; 38: daad034. 10.1093/heapro/daad034   37099680

⁷⁰ Thompson EL Preston SM Francis JKR. Social media perceptions and internet verification skills associated with human papillomavirus vaccine decision-making among parents of children and adolescents: cross-sectional survey. JMIR Pediatr Parent 2022; 5: e38297. 10.2196/38297   36103216

⁷¹ Shen AK Browne S Srivastava T Kornides ML Tan ASL. Trusted messengers and trusted messages: the role for community-based organizations in promoting COVID-19 and routine immunizations. Vaccine 2023; 41: 1994 - 2002. 10.1016/j.vaccine.2023.02.045   36803894

⁷² Hernandez RG Hagen L Walker K O’Leary H Lengacher C. The COVID-19 vaccine social media infodemic: healthcare providers’ missed dose in addressing misinformation and vaccine hesitancy. Hum Vaccin Immunother 2021; 17: 2962 - 4. 10.1080/21645515.2021.1912551   33890838

⁷³ Stecula DA Kuru O Jamieson KH. How trust in experts and media use affect acceptance of common anti-vaccination claims. Harv Kennedy Sch Misinformation Rev 2020; 10.37016/mr-2020-007

⁷⁴ US Centers for Disease and Prevention. About the vaccine conversations with parents campaign. 2019. https://www.cdc.gov/vaccines/hcp/conversations/about-vacc-conversations.html.

⁷⁵ National Academies. Communication strategies for building confidence in covid-19 vaccines: addressing variants and childhood vaccinations. https://nap.nationalacademies.org/catalog/26361/communication-strategies-for-building-confidence-in-covid-19-vaccines-addressing.

⁷⁶ Athey S Grabarz K Luca M Wernerfelt N. Digital public health interventions at scale: The impact of social media advertising on beliefs and outcomes related to COVID vaccines. Proc Natl Acad Sci U S A 2023; 120: e2208110120. 10.1073/pnas.2208110120   36701366

⁷⁷ Argyris YA Nelson VR Wiseley K Shen R Roscizewski A. Do social media campaigns foster vaccination adherence? A systematic review of prior intervention-based campaigns on social media. Telemat Inform 2023; 76: 101918. 10.1016/j.tele.2022.101918   36438457

⁷⁸ Sah P Vilches TN Moghadas SM. Return on investment of the covid-19 vaccination campaign in New York City. JAMA Netw Open 2022; 5: e2243127. 10.1001/jamanetworkopen.2022.43127   36409495

⁷⁹ Knudsen J Perlman-Gabel M Uccelli IG Jeavons J Chokshi DA. Combating misinformation as a core function of public health. NEJM Catal 2023; 4 10.1056/CAT.22.0198.

⁸⁰ Ruggeri K Palacios K Perkins ZA. Role of military forces in the New York State response to covid-19. JAMA Health Forum 2022; 3: e222136. 10.1001/jamahealthforum.2022.2136   36218979

⁸¹ Gallup. Confidence in institutions. 2007. https://news.gallup.com/poll/1597/Confidence-Institutions.aspx.

⁸² New York State Department of Health. New York State’s covid-19 vaccination program. 2020. https://www.governor.ny.gov/sites/default/files/atoms/files/NYS_COVID_Vaccination_Program_Book_10.16.20_FINAL.pdf

⁸³ Lewandowsky S, Cook J, Lombardi D. Debunking handbook 2020. 2020. https://digitalcommons.unl.edu/scholcom/245/.

⁸⁴ Ecker UKH Sharkey CXM Swire-Thompson B. Correcting vaccine misinformation: A failure to replicate familiarity or fear-driven backfire effects. PLoS One 2023; 18: e0281140. 10.1371/journal.pone.0281140   37043493

⁸⁵ Chan MS Albarracín D. A meta-analysis of correction effects in science-relevant misinformation. Nat Hum Behav 2023; 7: 1514 - 25. 10.1038/s41562-023-01623-8   37322236

⁸⁶ Ghezzi P Bannister PG Casino G. Online information of vaccines: information quality, not only privacy, is an ethical responsibility of search engines. Front Med (Lausanne) 2020; 7: 400. 10.3389/fmed.2020.00400   32850905

⁸⁷ Xin M Luo S Wang S. The roles of information valence, media literacy and perceived information quality on the association between frequent social media exposure and covid-19 vaccination intention. Am J Health Promot 2023; 37: 189 - 99. 10.1177/08901171221121292   35972385

⁸⁸ Hughes CT. Designing and implementing an educational social media campaign to increase HPV vaccine awareness among men on a large college campus. Am J Health Educ 2020; 51: 87 - 97 10.1080/19325037.2020.1722297

⁸⁹ Gursoy D Ekinci Y Can AS Murray JC. Effectiveness of message framing in changing COVID-19 vaccination intentions: Moderating role of travel desire. Tour Manag 2022; 90: 104468. 10.1016/j.tourman.2021.104468   34898791

⁹⁰ Faccin M Gargiulo F Atlani-Duault L Ward JK. Assessing the influence of French vaccine critics during the two first years of the COVID-19 pandemic. PLoS One 2022; 17: e0271157. 10.1371/journal.pone.0271157   35925962

⁹¹ Wang D Qian Y. Echo chamber effect in rumor rebuttal discussions about covid-19 in China: social media content and network analysis study. J Med Internet Res 2021; 23: e27009. 10.2196/27009   33690145

⁹² Müller J Tellier A Kurschilgen M. Echo chambers and opinion dynamics explain the occurrence of vaccination hesitancy. R Soc Open Sci 2022; 9: 220367. 10.1098/rsos.220367   36312563

⁹³ Kozyreva A Herzog SM Lewandowsky S. Resolving content moderation dilemmas between free speech and harmful misinformation. Proc Natl Acad Sci U S A 2023; 120: e2210666120. 10.1073/pnas.2210666120   36749721

⁹⁴ IPIE. Platform responses to misinformation: a meta-analysis of data. Jul 2023. https://www.ipie.info/research/sr2023-2.

⁹⁵ Aubin CS, Liedke J. Most Americans favor restrictions on false information, violent content online. Pew Research Center https://www.pewresearch.org/short-reads/2023/07/20/most-americans-favor-restrictions-on-false-information-violent-content-online/.

⁹⁶ Gavi, Unicef, World Health Organization, Vaccination Demand Hub, Health Enabled. Social listening: finding the signal through the noise. https://www.who.int/news/item/18-08- 2021-social-listening-finding-the-signal-through-the-noise. https://www.gavi.org/sites/default/files/2021-06/Finding-the-Signal-Through-the-Noise.pdf