Impact of vaccine pause due to Thrombosis with thrombocytopenia syndrome (TTS) following vaccination with the Ad26.COV2.S vaccine manufactured by Janssen/Johnson & Johnson on vaccine hesitancy and acceptance among the unvaccinated population

Abstract

Background

In response to reports of thrombosis with thrombocytopenia syndrome (TTS) post-vaccination, the Johnson & Johnson (J&J) vaccine was paused and then restarted in April 2021. Our objective was to assess whether this pause adversely impacted vaccine confidence.

Methods

Two large internet-based surveys were conducted in the US among adults to measure knowledge, attitudes and perceptions of the J&J vaccine pause and rates of vaccine hesitancy among unvaccinated persons before, during and after the pause.

Results

Among 66% of respondents aware of the pause, 44% identified blood clots as the reason for the pause without prompting. The impact of the pause on vaccine behavior among unvaccinated persons and perception of the vaccine safety system was mixed and modified by trust in the public health authorities. Those who were less willing to get vaccinated because of the pause were less inclined for all vaccines, not only the J&J product. Moreover, a notable proportion (22.1%) of the small number of persons (n = 30) vaccinated with the J&J vaccine after the pause reported not receiving information about the risk of TTS. The proportion of unvaccinated persons who were hesitant was increasing before and during the pause and then leveled off after the pause.

Conclusions

The J&J vaccine pause is unlikely to be a major barrier to vaccine uptake. Public attitudes about vaccines may be more resilient than appreciated, especially when safety issues are investigated with transparent communication. This paper has important implications for messaging and program administration with future vaccine-specific adverse events. Efforts may be warranted to ensure all persons being offered the J&J vaccine are made aware of the risk of TTS.

Introduction

Several vaccines received emergency use authorizations in the United States (US) beginning in December 2020, less than a year after China announced the first cases of COVID-19. Vaccine development was accelerated by decades of technologic advances coupled with massive government support. Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) vaccines hold great promise to end the pandemic. Less than 6 months after vaccine introduction, supply was outpacing demand in the US despite not achieving sufficient vaccine coverage to achieve herd immunity in all communities.

Attitudes toward COVID-19 disease and vaccines as well as vaccine coverage vary by age, ethnicity, education, political party affiliation, and region [1]. Concerns that the vaccines were rushed and not adequately studied for safety have been a major contributor to suboptimal vaccine acceptance [1, 2]. Adverse events following immunization, whether actually caused by vaccine or coincidental, can increase doubt among people who otherwise lean toward immunization and confirm doubts for those who lean against [2].

On April 13, the Centers for Disease Control and Prevention (CDC) and Food and Drug Administration (FDA) paused the use of the Janssen Ad26.COV2.S vaccine manufactured by Johnson & Johnson (J&J) in the US in response to reports of Thrombosis with thrombocytopenia syndrome (TTS) post-vaccination, primarily among women under 50 years of age. On April 23, following thorough review of available data, the agencies lifted the pause and use of J&J vaccine resumed.

While some criticized the pause as a distraction from the push for increased coverage, the pause and rigorous assessment demonstrated that vaccine safety was monitored, and signals were promptly investigated. With the objective of informing policy decisions around future safety issues, we set out to characterize the impact of the J&J vaccine pause on COVID-19 vaccine hesitancy in the general, unvaccinated population as well as age-, sex-, race-, and region-based subpopulations. We provide results from two large internet-based surveys to assess the impact of TTS on COVID-19 vaccine hesitancy.

Methods

Survey administration

For our primary analysis, we conducted and pooled data from two Web-based US surveys using RIWI’s (Real-Time Interactive World-Wide Intelligence) patented Random Domain Intercept Technology (RDIT) (S1 and S2 Appendices and https://www.protocols.io/view/impact-of-vaccine-pause-due-to-thrombosis-with-thr-cdbzs2p6). The vaccination intent survey was administered March 30—May 1, 2021. The rapid response (J&J) survey was administered April 30—May 13, 2021. We conducted the rapid response survey while implementing the vaccination intention survey and designed it specifically to continue capturing COVID-19 vaccine attitudes, beliefs, and behaviors while collecting additional cross-sectional information on attitudes and beliefs unique to the J&J pause event and immediately following its occurrence. We conducted a secondary descriptive analysis using data from the second survey on attitudes and beliefs unique to the J&J pause event to provide further context to the main analysis measuring the change in overall-, age-, sex-, race-, and region-specific vaccine hesitancy over time.

When internet users happen upon one of the hundreds of thousands of rotating domains that RIWI owns or controls (“dead links” that do not go to a functional website), these randomly engaged users are filtered through a series of proprietary algorithms to ensure that they are human respondents and subsequently invited to participate in a survey [3–6]. Respondents access surveys on all Web-enabled devices with internet access. Since all Web users have random chances of coming across RIWI surveys, respondents are geographically representative of internet users. The vaccination intent survey exposed over a million Web users to the survey landing page; of those, 78,697 opted-in to participate in the survey (6.8% opt-in rate). Over 600,000 Web users were exposed to the rapid response (J&J) survey, and 34,284 opted-in to participate (5.3% opt-in rate). The methodology collects the age, gender, and geo-location of respondents but no personally identifiable information. Respondents are informed that responses are fully anonymous and secure. No enticements are used; participants can exit at any point [7, 8]. Consent is implied by participant participation. This project was determined by Emory and JHU to be public health practice and not human subjects research.

Survey content

Both the vaccine intent and rapid response J&J surveys asked respondents if they had been vaccinated with at least one dose of a COVID-19 vaccine and, if not, asked them to select among scales of intention to get vaccinated. In addition, a variety of demographic and personal characteristics were collected along with questions about trusted sources of information and respondents’ reasons to get or not get vaccinated. In addition to assessing vaccination status and intention, we asked respondents to the J&J rapid response survey about their trust in the CDC and FDA, if they had heard about the J&J vaccine safety issue that caused the pause and, if so, what they had heard (free text), and if what they heard influenced their attitudes. Respondents were provided a brief description of the safety issue. Unvaccinated respondents were asked if the information about the J&J vaccine issue impacted their willingness to get the J&J vaccine or the available mRNA COVID-19 vaccines. All respondents were asked if the information impacted their trust in the vaccine safety monitoring systems.

Outcome of interest

The primary outcome variable of interest, denoted as Y_i (i = 30 March 2021–13 May 2021), was the daily proportion of unvaccinated survey participants who were hesitant or against getting vaccinated during three periods relative to the vaccination pause using the Johnson and Johnson (J&J) vaccine: 1) pre-pause (30 March– 12 April; 14 days); 2) during pause (13–23 April; 11 days); 3) after pause (24 April– 13 May; 21 days). The outcome variable was derived from a categorical variable based on a question asked to unvaccinated survey participants about their attitude toward COVID-19. When asked “what will you do when it is your turn to get the COVID-19 vaccine at no cost?”, participants selected from a list of categories: “I will…”:

* “…definitely get it as soon as I can”,

* “…likely get it as soon as I can”,

* “… likely get it but not right away”,

* “…likely not get vaccinated”, or

* “…definitely not get vaccinated”.

The dichotomization of the categorical term was based on the vaccine hesitancy elements described in the present body of literature, mainly confidence or acceptance of the COVID-19 vaccines as well as an element of time (e.g. delay) when getting vaccinated [9–11]. Those hesitant (delayed or unlikely) or against getting vaccinated were those who answered “likely but not right away”, “unlikely will”, and “definitely will not”. All proportions were weighted using survey design weights. We required at least 100 participants for any given day when calculating overall and stratified population proportions, and all proportions were weighted using design-based weights.

Covariates of interest

The main exposure of interest is the J&J pause with intervention time points defined as: 1) start of the J&J pause and 2) end of the J&J pause. We constructed dummy variables to indicate the different segmented periods (pre-intervention periods_1-2 versus post-intervention periods_1-2 using 0 and 1 codes, respectively). We looked at age and sex differences, stratifying by women (risk of TTS was reported among women) under the age of 50 years, women over the age of 50 years, and men. We explored race/ethnicity and COVID-19 vaccination coverage-based regional differences (25 June 2021 vaccination rank-based tertiles were determined to create regions indicating low, medium, and high relative coverage) by looking at daily proportions across sub-groups.

Statistical analysis

Analyses were performed using the two-sided significance level of 0.05. Weights were based on age and gender using the most recent national census data available (US Census Bureau 2021 projections) and were generated post-stratification using a raking algorithm. Chi-square tests were used to inspect the differences in the distribution of survey participant demographic characteristics, and p-values were calculated with a Rao-Scott second-order correction. Covariates missingness was evaluated and if greater than 5%, multiple imputation using 25 replications was conducted where missingness did not exceed ~30%. Data were combined from both surveys to ensure we had at least 100 participants for any given day when calculating overall and stratified population proportions, and all proportions were weighted [13].

Interrupted time series analysis estimated daily proportions of COVID-19 unvaccinated individuals who were COVID-19 vaccine hesitant or refusing pre-, during-, and post- J&J vaccine pause, using multiple segmented linear regression models [12]. Both surveys were required to carry out segmented regression. The level and slope parameters define the intercept and slope, respectively, for successive segments of the time series [11]. The level parameters were defined as the expected or average proportion of hesitant unvaccinated individuals at a specific time interval, and the slope parameters indicate the change in the proportion over a single unit of time (per day). Serial autocorrelation of error terms were tested using residual plots and by calculating the Durbin-Watson statistic using an estimate of close to 2.00 to rule out important autocorrelation [13]. The postestimation command estat hettest was used to test for heteroskedasticity to verify constant variance of all plots, [12, 14, 15] and rreg to look at possible exclusion of potential outliers. For the final model, autocorrelation was adjusted for by estimating the autocorrelation parameter using the command prais and including it in the model (Durbin-Watson statistics before adjustment (2.56); after adjustment: 2.09) [12, 14]. Weighted-based chi-square estimates were used to inspect differences in distributions of age, sex, race/ethnicity, and geographic region across vaccination intention categories during the entire study period. Data were analyzed using Stata (version 17, 2-core) for Mac OS [12].

Stratified analyses on race/ethnicity and regions were conducted based on COVID-19 vaccination coverage [16] (high, middle, and low coverage tertiles), and a combination of age and gender (women less than 50 years of age, women 50 years of age and older, and men). Models were constructed and checked using the same methods described above. Cross-model comparisons testing predictions and marginal effects were conducted using seemingly unrelated estimation (suest and the test commands for Wald test) and a robust variance estimator to adjust for duplication of daily observations in the stack dataset [17, 18].

A secondary analysis, designed to provide context for the trend in COVID-19 vaccine hesitant/refusing attitudes among the unvaccinated using a descriptive analysis of knowledge, attitudes, and beliefs about the pause was conducted with data from the rapid response J&J survey. We used the same methods for these descriptive statistics as those for the descriptive portion of the primary interrupted time series analysis as described above.

Results

Primary analysis

Descriptive analysis.

The proportion of unvaccinated individuals among all survey participants (n = 89,083) decreased from 68.0% to 64.1% to 61.1% during the pre-pause, during-pause, and post-pause time periods. Among the unvaccinated individuals (n = 56,193), distributions of age (p<0.01), sex (p<0.01), race/ethnicity (p<0.01), and region (p<0.01) differed significantly by vaccine hesitancy (Table 1). By survey (vaccine intent versus rapid response), statistically significant differences were not found by age (p = 0.30) or sex (p = 0.44) but did exist by race/ethnicity (p<0.01) and coverage-based region quintiles (p<0.01) (S1 Table). There were sex-, race/ethnicity-, and region-based differences by survey time period (each p<0.01) but no difference found by age (p = 0.07) (S2 Table).

[Figure omitted. See PDF.]

Impact of pause on vaccination intent among the unvaccinated.

The overall trend in daily proportions of hesitant individuals was increasing prior to the pause (0.7% per day, 95% CI 0.5–0.9% per day, p<0.01) and continued to increase during the pause (0.4% per day, 95% CI 0.1–0.7% per day, p = 0.01) (Fig 1 and Table 2); the pre-pause and during-pause trend difference was not statistically significant (p = 0.10). The change in the daily proportions of hesitant individuals stabilized following the end of the pause and showed no daily increase or decrease (0% per day, range -0.1%-0.1%, p = 0.94). The trend change following the end of the pause is significant (-0.4% per day, 95% CI -0.7%-0.1%, p = 0.010).

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

Increasing daily proportions of vaccine hesitancy before and during the pause and the tempered, level trend (about 0%) after the pause were not different when comparing women younger than 50 years to women 50 years and older (p = 0.52) and women younger than 50 years to men (p = 0.49, see Fig 1).

Associations between vaccine hesitancy and the pause were consistent among White, African American/Black, and Other races (p = 0.12) where there was an increase in hesitancy leading up to the pause and during the pause that leveled off after the pause (Fig 1). However, among Latinx, Asian and American Indian or Alaskan Native (AI/AN) groups, which showed different trends in hesitancy compared to the White racial group as well as the African American/Black and Other racial group (p<0.01 for both comparisons), there was a decreasing trend in hesitancy prior to the pause, which reversed direction and hesitancy increased during the pause, and then daily proportions of hesitancy recovered and leveled off after the pause.

No significant differences in trends between high-, middle-, and low- COVID-19 coverage-based regions (p = 0.960) were observed. All three regions showed significant trends before the start of the pause exhibiting 1.0% (p = 0.01), 0.6% (p<0.01), and 0.8% (p<0.01) daily increases in the high, middle, and low coverage regions, respectively. No other time period trends or changes in the trends were found to be significant.

Secondary analysis

Knowledge, attitudes, and beliefs.

Among the 34,284 adults who were rapid response J&J survey respondents, the majority (66%) were aware of the pause and blood clot formation was most commonly identified as the reason for the pause without prompting (41%). Of those citing clotting, only 12% related it to occurring primarily in women. White respondents were less likely than other races to be unaware of the pause (S3 Table).

Among unvaccinated respondents, 48.3% said that the information we provided about the pause made them less willing to get vaccinated with the J&J vaccine; 43.1% were less willing to get mRNA vaccines. Willingness to get vaccinated was modified by trust in CDC. Among those who did not trust CDC, 63.8% were less willing to get vaccinated with a different COVID-19 vaccine (Pfizer or Moderna) because of the pause compared with 22.4% of persons who trusted CDC. Similarly, 44.4% of persons trusting CDC were more likely to be vaccinated with a different COVID-19 vaccine (Pfizer or Moderna) because of the pause compared with 12.7% of persons who did not trust CDC. Decreased willingness to receive the J&J vaccine was slightly more evident among those ≥50 years of age (50.7%), 18–25 years (45.3%) and 30–49 years (47.4%; p = 0.05), and affected women slightly more than men (50.3% versus 46.3%; p = 0.03). Rural and Republican respondents were more likely to report the information about the pause made them less likely to be vaccinated compared with other respondents (S3 Table). The information we provided about the pause had no impact (40.1%) or increased trust (33.9%) in the vaccine safety monitoring system for the majority of respondents. The impact of information about the pause on trust in the safety system was also modified by trust in the CDC. Among respondents trusting the CDC, 46.6% reported the pause increased trust in the safety system, compared with 19.1% of persons who did not trust CDC. Similarly, 51.6% of persons not trusting CDC reported the information about the pause made them less likely to trust the safety system, compared with 13.3% of persons who trusted CDC. Of the persons previously vaccinated, 5.7% felt worse about their decision to get vaccinated after reading the information, and 30.7% felt better.

The majority of respondents who received the J&J vaccine since the pause (78.0%) reported that they discussed information about the J&J safety issue and/or received written information about the J&J safety issue when they were vaccinated. A substantial proportion of these respondents (22.1%) received no discussion or written materials. A minority of unvaccinated respondents (14.1%) expressed a preference for the J&J vaccine and 31.3% would get it if it was the only vaccine available for them. Nearly all these factors varied by race and political affiliation (S3 Table) and many also varied by gender, age, income, and region (full survey in S2 Appendix).

Discussion

We found that vaccine hesitancy generally increased from before and continuing through the J&J vaccination pause associated with the J&J vaccine adverse event but that related hesitancy leveled off after J&J immunization resumed, except for the subpopulation of Latinx, Asian, and AI/AN participants. Despite hesitancy leveling off, the end of the pause did not lead to a reduction in hesitancy. Important for messaging and program administration with future vaccine-specific adverse events, the associated concerns were related to all COVID-19 vaccines not just the J&J product, suggesting that adverse events in one vaccine can diminish trust in all. Nonetheless, perhaps demonstrating the value of transparent communication, the pause increased confidence in the vaccine safety system.

Even though TTS associated with this pause affected primarily younger women, the impact on daily trends in attitudes and intentions were not associated with gender or age suggesting that SARS CoV2 vaccine confidence is fragile across demographics. While there were differences in the impact of the pause on SARS CoV2 vaccine hesitancy by race and ethnicity, these were modest.

There are several important lessons from these findings. For the short term, the J&J vaccine pause is unlikely to be a major barrier to vaccine uptake for most unvaccinated persons. However, for a subset of the population, the pause may have reinforced existing concerns that the vaccines were developed too quickly and that additional safety problems may arise as the vaccines are used more broadly. Those who were already concerned about the safety of COVID-19 vaccines, particularly those who did not trust CDC, may have felt the pause validated their concerns. The net impact on vaccine acceptance was small. Reframing the pause as an example of the vaccine safety system functioning well may be useful in this subpopulation. In the long-term, public attitudes about vaccines may be more resilient than appreciated especially when safety issues are investigated with transparent communications to the public, even if doing so requires a pause in vaccination. This is the first study we are aware of that has been able to assess the impact of a vaccine safety issue on vaccine attitudes and intentions in real time. Retrospective review of the impact of vaccine safety scares are prone to many methodological challenges, typically including ecological rather than individual-level data [19].

There are several limitations to this study. The RIWI technology has many strengths but may be prone to non-response bias as all surveys. Although non-response bias is unknown, it is possible to assess trends in retention throughout a survey, which could yield further insights on the population of interest. While the methodology allows for an opt-in sample that is geographically representative of the population, it is possible for the sample to not be representative on other factors. To address this, we applied weights on age and gender in accordance with the United States Census. Of note, in our current data collection period (June 30—July 26, 2021), 72% of respondents (18 years of age and older) indicated that they have received at least one dose of the vaccine, which closely aligns with a contemporaneous CDC estimate of 68% [20]. All data including vaccine acceptance are self-reported. We used simple approaches to measuring some constructs, such as trust in FDA and CDC, whereas a longer survey could have used a more robust approach [13]. We found that among the 30 respondents who were vaccinated since the pause was lifted, 22.1% were not made aware of the risk of TTS which is concerning given the severity of TTS and the availability of alternative vaccines. However, these are self-reported data and the sample size is very small. In follow-up survey conducted immediately following this rapid survey (unpublished), we identified 1,845 persons who reported receiving the J&J vaccine after the pause; 17.5% reported no information about the risk of TTS was provided. Finally, we were not able to account for other factors that might impact vaccine hesitancy during this time period; however, this was the dominant story about the vaccine in the media and most people were aware of it.

Our study aligns with literature on the complexity of vaccine hesitancy. The end of a public vaccine safety event such as the J&J pause may have a tempering effect on the increasing proportion of vaccine hesitant attitudes among the remaining unvaccinated individuals over time—providing assurance that vaccine safety systems protect the public. Our study reinforces the importance of trust in public health authorities in response to a vaccine safety problem. Future research should focus on how trust in public health authorities can be improved and maintained. Public health authorities should be reassured that they can respond to a vaccine safety issue with transparency. A substantial proportion of the rapid response J&J survey respondents reported that they were not made aware of the risk of TTS at the time they were offered the J&J vaccine, though respondents may have under-reported or not recalled such warnings. While direct observation of the vaccination process would reduce this potential bias, efforts to ensure all persons being offered the J&J vaccine are made aware of the risk of TTS is warranted.

Supporting information

S1 Table. Sociodemographic characteristics by survey (Vaccination intention versus Rapid Response, J&J survey) (n = 54,727), weighted.

https://doi.org/10.1371/journal.pone.0274443.s001

(DOCX)

S2 Table. Sociodemographic characteristics by survey time period (pre-, during-, or post- JJ pause) among unvaccinated respondents (n = 54,727), weighted.

https://doi.org/10.1371/journal.pone.0274443.s002

(DOCX)

S3 Table. Rapid response J&J survey COVID-19 vaccine-related knowledge, attitudes, and beliefs by demographics, weighted.

https://doi.org/10.1371/journal.pone.0274443.s003

(DOCX)

S1 Appendix. Data description.

https://doi.org/10.1371/journal.pone.0274443.s004

(XLS)

S2 Appendix. Survey data.

https://doi.org/10.1371/journal.pone.0274443.s005

(DTA)

S1 Survey.

https://doi.org/10.1371/journal.pone.0274443.s006

(DOCX)

S2 Survey.

https://doi.org/10.1371/journal.pone.0274443.s007

(DOCX)

Citation: Salmon DA, Schuh HB, Sargent RH, Konja A, Harvey SA, Laurie S, et al. (2022) Impact of vaccine pause due to Thrombosis with thrombocytopenia syndrome (TTS) following vaccination with the Ad26.COV2.S vaccine manufactured by Janssen/Johnson & Johnson on vaccine hesitancy and acceptance among the unvaccinated population. PLoS ONE 17(10): e0274443. https://doi.org/10.1371/journal.pone.0274443

About the Authors:

Daniel A. Salmon

Roles: Conceptualization, Investigation, Methodology, Writing – original draft, Writing – review & editing

E-mail: [email protected]

Affiliations Institute for Vaccine Safety, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America, Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America, Department of Health, Behavior and Society, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America

https://orcid.org/0000-0002-1035-4422

Holly B. Schuh

Roles: Data curation, Formal analysis, Validation, Writing – review & editing

Affiliation: Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America

https://orcid.org/0000-0002-7901-4477

Rikki H. Sargent

Roles: Investigation, Software, Validation, Writing – review & editing

Affiliation: RIWI Corp., Toronto, Ontario, Canada

https://orcid.org/0000-0002-5247-0181

Alexis Konja

Roles: Investigation, Methodology, Writing – review & editing

Affiliation: Global Health Crisis Coordination Center (GHC3), Atlanta, Georgia, United States of America

Steven A. Harvey

Roles: Investigation, Methodology, Writing – review & editing

Affiliation: Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America

https://orcid.org/0000-0001-6936-0213

Shaelyn Laurie

Roles: Investigation, Software, Validation, Writing – review & editing

Affiliation: RIWI Corp., Toronto, Ontario, Canada

Brandy S. Mai

Roles: Investigation, Methodology, Writing – review & editing

Affiliation: Global Health Crisis Coordination Center (GHC3), Atlanta, Georgia, United States of America

Leo F. Weakland

Roles: Investigation, Methodology, Writing – review & editing

Affiliation: Global Health Crisis Coordination Center (GHC3), Atlanta, Georgia, United States of America

James V. Lavery

Roles: Investigation, Methodology, Writing – review & editing

Affiliations Hubert Department of Global Health, Rollins School of Public Health, Emory University, Atlanta, Georgia, United States of America, Center for Ethics, Emory University, Atlanta, Georgia, United States of America

Walter A. Orenstein

Roles: Conceptualization, Methodology, Writing – original draft, Writing – review & editing

Affiliations Hubert Department of Global Health, Rollins School of Public Health, Emory University, Atlanta, Georgia, United States of America, School of Medicine, Emory University, Atlanta, Georgia, United States of America

Robert F. Breiman

Roles: Conceptualization, Funding acquisition, Investigation, Methodology, Writing – original draft, Writing – review & editing

Affiliations Hubert Department of Global Health, Rollins School of Public Health, Emory University, Atlanta, Georgia, United States of America, School of Medicine, Emory University, Atlanta, Georgia, United States of America

References

1. Kaiser Family Foundation. KFF COVID-19 Vaccine Monitor. https://www.kff.org/coronavirus-covid-19/dashboard/kff-covid-19-vaccine-monitor-dashboard/. Accessed 25 August 2021.

2. Salmon DA, Dudley MZ, Brewer J, et al. COVID-19 vaccination attitudes, values and intentions among United States adults prior to emergency use authorization. Vaccine, 2021; 39: 2698–2711. pmid:33781601

3. Seeman N, Seeman MV. An anonymous survey method for obtaining sensitive, personal, or embarrassing information in the context of mental illness stigma. SAGE Research Methods Cases: Medicine and Health. 2020.

4. Seeman N. Use data to challenge mental health stigma. Nature, 2015; 28: 309. pmid:26672520

5. Seeman N, Ing A, Rizo C. Assessing and responding in real time to online anti-vaccine sentiment during a flu pandemic. Healthc Q, 2010; 13 Spec No: 8–15. pmid:20959725

6. Seeman N, Tang S, Brown AD, Ing A. World survey of mental illness stigma. J Affect Disord, 2016; 190: 115–121. pmid:26496017

7. Roder-DeWan S, Gage A, Hirschhorn LR, et al. Level of confidence in and endorsement of the health system among internet users in 12 low-income and middle-income countries. BMJ Glob Health, 2020; 5(8): e002205. pmid:32859647

8. Roder-DeWan S, Gage AD, Hirschhorn LR, et al. Expectations of healthcare quality: A cross-sectional study of internet users in 12 low- and middle-income countries. PLoS Med, 2019; 16(8): e1002879. pmid:31390364

9. Hickler B, MacDonald NE, Senouci K, Schuh HB; informal Working Group on Vaccine Demand (iWGVD); Strategic Advisory Group of Experts on immunization (SAGE) Working Group on Decade of Vaccines. Efforts to monitor Global progress on individual and community demand for immunization: Development of definitions and indicators for the Global Vaccine Action Plan Strategic Objective 2. Vaccine. 2017 Jun 16;35(28):3515–3519. Epub 2017 May 20. pmid:28536028

10. Singh P, Dhalaria P, Kashyap S, Soni GK, Nandi P, Ghosh S, et al. Strategies to overcome vaccine hesitancy: a systematic review. Syst Rev. 2022 Apr 26;11(1):78. pmid:35473819.

11. Dubé E, Gagnon D, Nickels E, Jeram S, Schuster M. Mapping vaccine hesitancy-Country-specific characteristics of a global phenomenon. Vaccine. 2014;32(49):6649–54. pmid:25280436

12. Tzeng I, Chien K, Tu Y, et al. Segmented regression analysis of emergency departments patient visits from Septicemia in Taiwan. Heal Policy Technol, 2018; 7: 149–155. pmid:32289003

13. Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther, 2002; 27: 299–309. pmid:12174032

14. Williams R. Heteroskedasticity. 2020. https://www3.nd.edu/~rwilliam/stats2/l25.pdf. Accessed 25 August 2021.

15. Stata Manual. https://www.stata.com/manuals/rrreg.pdf. Accessed 25 August 2021.

16. See How Vaccinations Are Going in Your County and State. New York Times. https://www.nytimes.com/interactive/2020/us/covid-19-vaccine-doses.html. Accessed 25 August 2021.

17. Mize TD, Doan L, Long JS. A General Framework for Comparing Predictions and Marginal Effects across Models. Sociological Methodology, 2019; 49: 152–189.

18. Weesie J. Seemingly Unrelated Estimation and The Cluster-adjusted Sandwich Estimator. Stata Technical Bulletin, 1999; 52: 34–47, Reprinted in Stata Technical Bulletin Reprints, 2000; 9: 231–248.

19. Gangarosa EJ, Galazka AM, Wolfe CR, et al. Impact of anti-vaccine movements on pertussis control: the untold story. Lancet, 1998; 351: 356–361. pmid:9652634

20. Centers for Disease Control and Prevention. COVID Data Tracker. https://covid.cdc.gov/covid-data-tracker/#vaccinations. Accessed 25 August 2021.