Development and validation of the Oxford Pandemic Attitude Scale-COVID-19 (OPAS-C): an internet-based cross-sectional study in the UK and USA

Abstract

Objectives

To develop and validate the Oxford Pandemic Attitude Scale-COVID-19 (OPAS-C), a multidimensional scale that addresses seven domains over 20 items including stress, fear, loneliness, sense of community, belief that the pandemic is a hoax or exaggerated, the use of and attitude to non-pharmaceutical interventions and vaccine hesitancy, in a single measure.

Design

Cross-sectional validation study.

Setting

Internet based with respondents in the USA and UK.

Participants

General community respondents using the Prolific Academic platform.

Main outcome measures

Exploratory factor analyses with promax oblique rotation and confirmatory factor analysis including goodness of fit indices: root mean square error of approximation (RMSEA), standardised root mean square residual (SRMR) and comparative fit index (CFI). Reliability as internal consistency using Cronbach’s alpha. Convergent and discriminant validity using Pearson correlation coefficients.

Results

The sample included 351 respondents in the USA and the factorial structure was confirmed using a separate set of 348 respondents in the UK. The OPAS-C had excellent goodness of fit characteristics, with an RMSEA of 0.047 (90% CI 0.037 to 0.056), SRMR of 0.043 and CFI of 0.962. Reliability was excellent, demonstrating Cronbach’s alpha of 0.87 in both the US and UK samples. Convergent validity showed correlation coefficients of 0.54 and 0.49 in the US and UK samples, respectively. Discriminant validity demonstrated correlations of 0.21 and 0.26 in the US and UK samples, respectively.

Conclusions

The OPAS-C represents the first validated scale that addresses mental health measures and public health-relevant responses to COVID-19, and may be a useful measure for use in future longitudinal and cross-sectional studies. Further international validation beyond the USA and UK may be helpful.

Correspondence to Dr Jonathan Kantor; [email protected]

Strengths and limitations of this study

• Inclusion of a range of stakeholders, including the general public, in the creation of the Oxford Pandemic Attitude Scale-COVID-19 (OPAS-C) adds to the clinical meaningfulness and validity of the scale.

• Encompassing a wide range of domains relevant to medical and public health researchers in a single scale may permit feasible, and therefore cost-effective, longitudinal and cross-cultural assessments.

• The excellent fit characteristics of the factorial structure and the robustness of these findings across populations mean that the OPAS-C is generalisable across the populations studied.

• Internet-based sampling may affect generalisability, and biases—such as response bias and social desirability bias—may have affected our findings.

• Determining the sensitivity to change and test–retest reliability of the OPAS-C may be helpful in the future.

Introduction

Pandemic COVID-19 has taken a significant toll on the lives, liberties and livelihoods of people across the world, and understanding public attitudes to the disease is a high priority for public health researchers and medical and mental health experts.^1–3 Public attitudes to COVID-19 and the responses of the general public to the pandemic, social distancing and other mitigation measures, and other restrictions may have a direct bearing on the success of public health efforts.^4–6

While numerous studies have individually evaluated public attitudes to the COVID-19 pandemic,^{1 7–13} in order to better compare these attitudes, actions and beliefs over time and across geographic locations, validated scales provide greater insight and allow for more meaningful longitudinal and geographic cross-sectional comparisons.¹⁴ One of the focuses of the pandemic has been on the geographic heterogeneity in general public attitudes and behaviours across countries—and this variability has sometimes been associated with differential outcomes.¹⁵ Several scales have already been developed for selected public responses to COVID-19, including those that focus on fear,^{16 17} anxiety,¹⁸ stress (including distress, depression, anxiety and perception of danger),¹⁹ fatalism,²⁰ knowledge,²¹ shopping behaviour²² and hope.²³ While these scales are all important tools for researchers and clinicians, none of them were designed for public health researchers to address the attitudes and behaviours, positive and negative emotions and perceptions of vaccines and community that directly affect global health considerations.

We therefore aimed to develop and validate a comprehensive multidimensional scale that would assess the general public’s response to COVID-19 in its myriad dimensions, from fear, stress and loneliness through to positive responses such as sense of community, while also addressing key behavioural considerations that are vitally important to public health efforts, such as adherence and attitude to non-pharmaceutical interventions (NPIs) and vaccine hesitancy. Given the dynamic nature of the pandemic, the goal was to develop a short feasible scale so that it does not place an undue burden on respondents or researchers.

Methods

Item generation

The Oxford Pandemic Attitude Scale-COVID-19 (OPAS-C) was developed following established psychometric protocols and best practices.¹⁴ Domain generation was performed via an iterative deductive and inductive approach. After a literature review, online focus groups with members of the public were used to develop a set of possible domains of interest. A group of expert reviewers in the fields of medicine and public health were also consulted regarding domain inclusion in keeping with standard scale development methodology.¹⁴ Domains of interest included (1) acute stress; (2) fear of contracting COVID-19; (3) anxiety; (4) loneliness; (5) use of and attitude to NPIs; (6) belief that the pandemic is a hoax or exaggerated; (7) sense of community and optimism; and (8) vaccine hesitancy. All responses were recorded on a 5-point Likert scale ranging from strongly agree to strongly disagree. Items were reverse scored, as appropriate, to yield a scale where higher values represented greater levels of concern regarding COVID-19. All subjects provided informed consent for participation and were permitted to withdraw from their anonymous surveys at any time. Statistical analyses were performed using Stata for Mac V.13 (Stata, College Station, Texas, USA) and Lavaan V.0.6-8 (University of Ghent, Belgium).

Data collection

Two separate samples were used in the development of the OPAS-C. First, a sample of 350 respondents in the USA were recruited through Prolific Academic (Oxford, UK), an academic survey panel site, and a draft scale was presented using the Qualtrics platform (Qualtrics, Provo, Utah, USA). All respondents were rewarded with a small payment (<£1) and were offered the possibility of withdrawing their consent for anonymous participation at any time. This sample was used to develop the scale and perform exploratory factor analysis, as detailed below. A second sample of respondents from the UK was then solicited in a similar fashion using the same electronic survey recruitment tools. This sample was used for confirmatory factor analysis. Results from the USA and UK were evaluated separately to establish the robustness of the scale. Demographic information was self-reported by respondents and was linked with responses through an anonymous 25-digit alphanumeric key. Baseline information included age, sex, employment status, education history and household income.

Domain and item reduction and exploratory factor analysis

Maximum likelihood parallel analysis was performed to help determine the appropriate number of domains to retain for the overall scale.²⁴ This approach is more conservative than the reliance on eigenvalues over 1, and a recent study has highlighted the importance of relying on more than one approach when deciding on the number of factors to retain.^{25 26} Item reduction was approached by examining polychoric correlations (item–item correlations) and polyserial correlations (item–test correlations). Though low correlations (<0.3) may suggest consideration of deletion, we decided a priori that since the broadly multidimensional scale includes domains that would be expected to correlate poorly, then the potential domain-relevant contributions of the item may supersede the absolute correlation values.²⁷ Exploratory factor analysis was performed using maximum likelihood estimation followed by oblique factor rotation.

Confirmatory factor analysis

The robustness of the findings from the US sample was tested by performing confirmatory maximum likelihood factor analysis using structural equation modelling on the UK sample with the model developed in the US sample using maximum likelihood estimation with robust SEs and a Satorra-Bentler scaled test statistic. Goodness of fit indices including the root mean square error of approximation (RMSEA), the standardised root mean square residual (SRMR) and the comparative fit index (CFI) were assessed to gauge the model fit. Values of RMSEA ≤0.06, SRMR ≤0.08 and CFI ≥0.95 suggest an excellent fit.²⁸

Reliability and validity

Reliability as internal consistency was assessed by examining Cronbach’s alpha for the OPAS-C. An ideal scale has high internal consistency, though when correlations are very high, this may suggest the possibility of redundancy; therefore, the ideal range for Cronbach’s alpha is between 0.75 and 0.9.²⁹

Convergent validity was assessed by examining the correlation between the OPAS-C and agreement with a single question addressing the respondents’ overall response to the pandemic (‘thinking about the pandemic makes me feel anxious’) in both the US and UK samples. Discriminant validity was assessed by the correlation between the OPAS-C score and agreement with a non-specific statement (‘I think we are all in this together’) in the US and UK samples. A cut-off of 0.3 was used for these correlations, so that values >0.3 are desirable for convergent validity and values <0.3 are desirable for discriminant validity, consistent with prior studies.¹⁹

Patient and public involvement

This study was designed to develop a scale that could be used to measure public responses to COVID-19. As such, online anonymous focus groups were used to aid in domain generation and public responses were solicited through Prolific Academic to help develop the scale from the time of inception, and the goal of the research was to reflect the priorities and preferences of the public. Results will be disseminated and hopefully lead other researchers to use the OPAS-C in assessing public responses to the COVID-19 pandemic.

Results

Respondent characteristics

For the US sample, of the 359 respondents who started the survey, 351 subjects returned the finished surveys, yielding a 97.8% completion rate in the survey panel. The mean (SD) age was 30 (10.5), and respondents ranged in age from 18 to 72. 46.8% of respondents were male, and the median (IQR) household income was $50 000 (26 000, 95 000) for those who were working. A total of 26.3% of respondents were not currently working; 15.2% of respondents were high school graduates and 49.1% had university degrees.

In the UK sample, of the 354 respondents who started the survey, 348 subjects returned the finished surveys, yielding a 98.3% completion rate in the survey panel. The mean (SD) age was 30.5 (11.3), and respondents ranged in age from 18 to 71. 51.4% of respondents were male, and the median (IQR) household income was £29 550 (16 750, 48 500) for those who were working. A total of 20.6% of respondents were not currently working; 20.1% of respondents were high school graduates and 37.9% had university degrees.

Domain and item reduction and exploratory factor analysis

A total of 50 items were developed which were reduced to 43 prior to testing; the large item pool was designed to test multiple similar iterations of items in a focus group format. We included between 4 and 10 items per domain a priori for consideration, depending on the clinical breadth of the domain. Maximum likelihood parallel analysis suggested a 7-factor solution. A total of 23 items were deleted due to low polychoric and polyserial correlations when clinically relevant, unifactorial outcomes or redundancy. Factor loadings from the 20 retained items after maximum likelihood exploratory factor analysis and promax oblique rotation are presented in table 1. The eigenvalues for these factors were 3.97, 2.49, 1.69, 1.12, 1.40, 1.25 and 1.48. This 7-factor solution reflected a simple structure with appropriate loading. Factors correlated with the domains of (1) acute stress; (2) fear; (3) loneliness; (4) sense of community; (5) belief that the pandemic is a hoax or exaggerated; (6) use of and attitude to NPIs; and (7) vaccine hesitancy. Anxiety as a stand-alone domain was culled due to unifactorial loading and significant overlap with the more general stress domain. The final scale and scoring is presented in table 2 and factor intercorrelations are presented in table 3.

Table 1

Factor loadings in the US sample exploratory factor analysis

Item	I Stress	II Fear	III Loneliness	IV Community	V Exaggerated	VI NPIs	VII Vaccine
I am having trouble relaxing because of the virus.	0.81	−0.03	0.01	−0.09	0.05	0.01	−0.02
I cannot control worrying about the virus.	0.78	0.03	0.02	−0.10	0.02	−0.01	−0.01
I think about the virus more than I would like.	0.92	0.07	−0.10	−0.02	0.09	−0.01	−0.04
Thoughts of the virus pop into my head even when I do not want them to.	0.81	0.06	0.03	−0.02	−0.04	−0.03	−0.03
I have trouble concentrating because I think about the virus so much.	0.92	−0.07	−0.08	0.08	−0.08	0.02	−0.06
I check the news or online sources for updates on the virus more than I would like.	0.66	0.07	0.12	0.03	0.04	−0.05	0.14
I am having trouble sleeping because I am thinking about the virus.	0.75	−0.01	−0.01	0.06	−0.07	−0.02	0.04
I am afraid of getting the virus myself.	0.07	0.92	−0.04	−0.01	−0.03	0.03	0.08
I am afraid of a family member getting the virus.	0.03	0.99	0.01	0.04	0.01	0.03	−0.01
I feel isolated from other people in the pandemic.	0.08	0.31	0.36	0.04	0.03	−0.07	−0.05
With the pandemic, I feel like I cannot connect to other people.	0.19	−0.04	0.50	−0.17	0.02	0.01	0.02
I feel close to other people.	0.00	0.09	−0.01	0.97	0.03	−0.06	0.06
I feel part of a larger community of people.	0.06	−0.08	0.07	0.61	0.00	0.06	0.04
I think the pandemic is a hoax.	0.000	−0.04	−0.06	0.01	0.88	−0.04	0.04
I think people are getting too excited about the pandemic.	−0.01	−0.03	−0.04	0.03	0.71	0.11	0.03
I am wearing a face covering or mask when I am around people.	0.02	0.05	0.02	0.05	−0.15	0.81	0.00
I am social distancing.	−0.06	−0.04	−0.01	−0.09	0.04	0.81	−0.01
I am washing my hands frequently.	0.07	0.02	−0.01	0.08	0.13	0.50	0.01
I would take the coronavirus vaccine when it becomes available.	−0.02	0.09	0.01	0.02	0.03	−0.05	0.89
I would have my children or parents take the vaccine when it comes out.	−0.03	0.03	0.00	0.10	0.01	0.05	0.92

Significant (>0.35) factors are bolded.

NPI, non-pharmaceutical intervention.

Table 2

The OPAS-C

No	Item	Domain
1	I am having trouble relaxing because of the virus.	Stress
2	I cannot control worrying about the virus.	Stress
3	I think about the virus more than I would like.	Stress
4	Thoughts of the virus pop into my head even when I do not want them to.	Stress
5	I have trouble concentrating because I think about the virus so much.	Stress
6	I check the news or online sources for updates on the virus more than I would like.	Stress
7	I am having trouble sleeping because I am thinking about the virus.	Stress
8	I am afraid of getting the virus myself.	Fear
9	I am afraid of a family member getting the virus.	Fear
10	I feel isolated from other people in the pandemic.	Loneliness
11	With the pandemic, I feel like I cannot connect to other people.	Loneliness
12	I feel close to other people.	Community
13	I feel part of a larger community of people.	Community
14	I think the pandemic is a hoax.	Exaggerated
15	I think people are getting too excited about the pandemic.	Exaggerated
16	I am wearing a face covering or mask when I am around people.	NPIs
17	I am social distancing.	NPIs
18	I am washing my hands frequently.	NPIs
19	I would take the coronavirus vaccine when it becomes available.	Vaccine
20	I would have my children or parents take the vaccine when it comes out.	Vaccine

All answer choices are rated using a 5-point Likert scale (strongly disagree through strongly agree). For questions 1–13, strongly agree is scored as 5, while for questions 14–20 strongly agree is scored as 1 (reverse scoring). Higher values reflect a greater burden and the total score therefore ranges from 20 to 100.

NPI, non-pharmaceutical intervention; OPAS-C, Oxford Pandemic Attitude Scale-COVID-19.

Table 3

Factor intercorrelations

	I	II	III	IV	V	VI	VII
I	1
II	0.436	1
III	0.369	0.474	1
IV	0.138	0.298	0.461	1
V	0.309	0.303	0.441	0.399	1
VI	0.277	0.412	0.590	0.462	0.382	1
VII	0.315	0.257	0.236	0.420	0.228	0.376	1

Confirmatory factor analysis

Confirmatory factor analysis was first performed on the UK sample using the model developed in the US sample. Using maximum likelihood estimation with robust SEs and a Satorra-Bentler scaled test statistic, the 7-factor model including 20 items demonstrated excellent goodness of fit characteristics, with RMSEA=0.047 (90% CI 0.037 to 0.056), SRMR=0.043 and CFI=0.962.

The total OPAS-C scores were normally distributed in both the US and UK populations as demonstrated by the results from the Shapiro-Wilk test (p>Z=0.31 and 0.30 for the US and UK populations, respectively). Mean and 95% CI for the OPAS-C were 54.7 (53.5 to 56.0) in the US population and 56.1 (54.9 to 57.2) in the UK population. The similarities in OPAS-C distributions between the two populations further bolster the robustness of the scale.

Reliability and validity

Cronbach’s alpha for the overall OPAS-C was 0.87 in both the US and UK samples, suggesting excellent reliability as measured by internal consistency in both populations. Reliability for each subscale ranged from 0.66 to 0.93 in the US sample, as outlined in table 4. Convergent validity, as assessed by the correlation between the OPAS-C and agreement with a single question addressing the overall response to the pandemic (‘thinking about the pandemic makes me feel anxious’), was good at 0.54 and 0.49 in the US and UK samples, respectively. Discriminant validity, as assessed by the correlation between the OPAS-C score and agreement with a non-specific statement (‘I think we are all in this together’), was also good, with values of 0.21 and 0.26 in the US and UK samples, respectively.

Table 4

Subscale reliability, as measured by Cronbach’s alpha

Subscale	US sample	UK sample
Stress	0.92	0.90
Fear	0.77	0.75
Loneliness	0.74	0.69
Community	0.77	0.72
Exaggerated	0.66	0.51
NPIs	0.77	0.72
Vaccine	0.93	0.93

NPI, non-pharmaceutical intervention.

Discussion

We found that the OPAS-C, the first comprehensive validated multidimensional scale assessing public attitudes and responses to the COVID-19 pandemic, was robust across both US and UK populations and performed well on goodness of fit indices as well as in reliability and validity testing. The robustness of the factorial structure of the scale, coupled with its excellent fit and broad domain structure, as well as its outstanding feasibility, suggests that the OPAS-C may represent a useful addition in the public health response to COVID-19.

Several other scales have been previously validated to assess public responses to the COVID-19 pandemic, but most of these are either unidimensional or address only psychological responses. Therefore, while these scales are excellent for their intended purpose, our interest was in globally evaluating attitudes, actions and beliefs and developing a single measure that may be useful across populations and over different timeframes for assessing the public health burden and response to COVID-19. The OPAS-C does not claim to be superior to any other validated scale; the aim of the scale, however, and its scope, are different from those that have been previously developed in that it is expressly designed to include the key information that public health researchers need—while also perhaps eliding over some of the details that would be important to those studying only mental health outcomes. Given the burden that the COVID-19 pandemic has already placed on health and economic systems worldwide,³⁰ there is a clear need for a scale that can be easily deployed and is feasible (the 20-item OPAS-C can be completed in under 3 min) and that can capture myriad responses to the pandemic.

Additional strengths of this validation study include the use of both US and UK populations for the validation process and the broad inclusion of societal responses, from stress and fear to sense of community and vaccine hesitancy. Including various stakeholders, such as the general public, as well as researchers, is an important aspect of scale development, and helps ensure that the scale has adequate content validity.^{31 32} Moreover, the excellent fit characteristics of our factorial structure and the robustness of these findings across populations suggest that the OPAS-C may be a valuable adjunct for public health and social science researchers alike.

There are several limitations to this work. First, the sampling used in developing and validating the OPAS-C may not be representative of the overall population of interest. The robustness of the factorial structure of the scale and the outcomes overall—the US and UK samples differed by only 1.4 points out of a total possible score of 100—suggests that the generalisability of the OPAS-C is likely adequate. Further validation in multiple countries and different languages may be helpful to better establish the generalisability of this scale beyond the USA and UK. Second, further convergent and discriminant validity assessments could be considered using other established scales; since the OPAS-C is unique in its broad multifactorial structure, however, this would require the inclusion of multiple scales and additional questions. Moreover, the convergent and discriminant validity seen using single-question comparators support the validity of the OPAS-C. Still, since these rely on single-question comparators this can be regarded as preliminary validity evidence. Third, the internal consistency reliability of some of the subscales was limited—particularly for the items regarding pandemic exaggeration—and this may be a function of both having only two items included in the subscale and the discordance between the potentially polarising word ‘hoax’ and more general concerns regarding whether concerns have been exaggerated; future research in this area may be warranted. Fourth, we did not evaluate the test–retest reliability of the OPAS-C or its sensitivity to change over time. Future work that addresses the sensitivity of the OPAS-C to change as attitudes and behaviours evolve will be helpful. Finally, as with any study that relies on online data collection, biases such as response bias and social desirability bias may affect the results, though the replicability and robustness of our findings across disparate populations suggest that the OPAS-C is a valid and reliable scale for gaining valuable public health insight into pandemic COVID-19, and may serve as a useful addition to the burgeoning array of valid and clinically meaningful scales that can be used in studying COVID-19.

Data availability statement

Data are available upon reasonable request. Data are available from the corresponding author.

Ethics statements

Patient consent for publication

Not required.

Ethics approval

This research was approved by the Ascension Health Institutional Review Board through exempted approval as no identifying information was obtained from anonymous participants (approval 1580959-1).

Footnote

Contributors Study conception: JK. Study design: JK, BNK. Statistical analyses: JK, BNK. Writing: JK, BNK. Oversight: JK.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

Provenance and peer review Not commissioned; externally peer reviewed.

References

¹ Kantor BN, Kantor J. Non-pharmaceutical interventions for pandemic COVID-19: a cross-sectional investigation of US General public beliefs, attitudes, and actions. Front Med 2020; 7: 384. doi:10.3389/fmed.2020.00384 http://www.ncbi.nlm.nih.gov/pubmed/32719807

² Geldsetzer P. Knowledge and perceptions of COVID-19 among the general public in the United States and the United Kingdom: a cross-sectional online survey. Ann Intern Med 2020; 173: 157–60. doi:10.7326/M20-0912 http://www.ncbi.nlm.nih.gov/pubmed/32196071

³ Kantor BN, Kantor J. Mental health outcomes and associations during the COVID-19 pandemic: a cross-sectional population-based study in the United States. Front Psychiatry 2020; 11: 569083. doi:10.3389/fpsyt.2020.569083 http://www.ncbi.nlm.nih.gov/pubmed/33424655

⁴ Alwan NA, Bhopal R, Burgess RA, et al. Evidence informing the UK’s COVID-19 public health response must be transparent. The Lancet 2020; 395: 1036–7. doi:10.1016/S0140-6736(20)30667-X

⁵ Baral SD, Mishra S, Diouf D, et al. The public health response to COVID-19: balancing precaution and unintended consequences. Ann Epidemiol 2020; 46: 12–13. doi:10.1016/j.annepidem.2020.05.001 http://www.ncbi.nlm.nih.gov/pubmed/32532367

⁶ Scally G, Jacobson B, Abbasi K. The UK's public health response to covid-19. BMJ 2020; 369: m1932. doi:10.1136/bmj.m1932 http://www.ncbi.nlm.nih.gov/pubmed/32414712

⁷ Seale H, Heywood AE, Leask J, et al. COVID-19 is rapidly changing: examining public perceptions and behaviors in response to this evolving pandemic. PLoS One 2020; 15: e0235112. doi:10.1371/journal.pone.0235112 http://www.ncbi.nlm.nih.gov/pubmed/32574184

⁸ Abdelhafiz AS, Mohammed Z, Ibrahim ME, et al. Knowledge, Perceptions, and Attitude of Egyptians Towards the Novel Coronavirus Disease (COVID-19). J Community Health 2020; 45: 881–90. doi:10.1007/s10900-020-00827-7

⁹ Al-Hanawi MK, Angawi K, Alshareef N, et al. Knowledge, attitude and practice toward COVID-19 among the public in the Kingdom of Saudi Arabia: a cross-sectional study. Front Public Health 2020; 8: 217. doi:10.3389/fpubh.2020.00217 http://www.ncbi.nlm.nih.gov/pubmed/32574300

¹⁰ Azlan AA, Hamzah MR, Sern TJ, et al. Public knowledge, attitudes and practices towards COVID-19: a cross-sectional study in Malaysia. PLoS One 2020; 15: e0233668. doi:10.1371/journal.pone.0233668 http://www.ncbi.nlm.nih.gov/pubmed/32437434

¹¹ Cowling BJ, Ali ST, Ng TWY, et al. Impact assessment of non-pharmaceutical interventions against coronavirus disease 2019 and influenza in Hong Kong: an observational study. Lancet Public Health 2020; 5: e279–88. doi:10.1016/S2468-2667(20)30090-6 http://www.ncbi.nlm.nih.gov/pubmed/32311320

¹² Lin Y, Hu Z, Alias H, et al. Knowledge, attitudes, impact, and anxiety regarding COVID-19 infection among the public in China. Front Public Health 2020; 8. doi:10.3389/fpubh.2020.00236

¹³ Maher PJ, MacCarron P, Quayle M. Mapping public health responses with attitude networks: the emergence of opinion‐based groups in the UK’s early COVID‐19 response phase. Br J Soc Psychol 2020; 59: 641–52. doi:10.1111/bjso.12396

¹⁴ Boateng GO, Neilands TB, Frongillo EA, et al. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health 2018; 6: 149. doi:10.3389/fpubh.2018.00149 http://www.ncbi.nlm.nih.gov/pubmed/29942800

¹⁵ Miller LE, Bhattacharyya R, Miller AL. Spatial analysis of global variability in Covid-19 burden. Risk Manag Healthc Policy 2020; 13: 519–22. doi:10.2147/RMHP.S255793 http://www.ncbi.nlm.nih.gov/pubmed/32581614

¹⁶ Ahorsu DK, Lin C-Y, Imani V, et al. The fear of COVID-19 scale: development and initial validation. Int J Ment Health Addict 2020; 1. doi:10.1007/s11469-020-00270-8

¹⁷ Winter T, Riordan BC, Pakpour AH, et al. Evaluation of the English version of the fear of COVID-19 scale and its relationship with behavior change and political beliefs. Int J Ment Health Addict 2020: 1–11. doi:10.1007/s11469-020-00342-9 http://www.ncbi.nlm.nih.gov/pubmed/32837431

¹⁸ Viswa Chaitanya C, Srinivas P, Viswanath V. Development and initial validation of the COVID-19 anxiety scale. Indian J Public Health 2020; 64: 201–4. doi:10.4103/ijph.ijph_492_20

¹⁹ Taylor S, Landry CA, Paluszek MM, et al. Development and initial validation of the COVID stress scales. J Anxiety Disord 2020; 72: 102232. doi:10.1016/j.janxdis.2020.102232 http://www.ncbi.nlm.nih.gov/pubmed/32408047

²⁰ Christian RM, Alarcon JFR-, Macarena C. Fatalism due to the possibility of coronavirus infection: generation and validation of an instrument (F-COVID-19). Kasmera 2020; 48: e48118032020. doi:10.5281/zenodo.3732353

²¹ Christian RM, Rodriguez-Alarcon JF, Macarena C. Validation of a short scale for measuring the level of basic knowledge about coronavirus, Peru (KNOW-P-COVID-19). Kasmera 2020; 48: e48106042020. doi:10.5281/zenodo.3827988

²² Szymkowiak A, Kulawik P, Jeganathan K. In-store epidemic behavior: scale development and validation, 2020.

²³ Bernardo ABI, Mendoza NB. Measuring hope during the COVID-19 outbreak in the Philippines: development and validation of the state locus-of-hope scale short form in Filipino. Curr Psychol 2020: 1–10. doi:10.1007/s12144-020-00887-x 32837128

²⁴ Dinno A. Implementing horn’s parallel analysis for principal component analysis and factor analysis. Stata J 2009; 9: 291–8. doi:10.1177/1536867X0900900207

²⁵ Horn JL. A rationale and test for the number of factors in factor analysis. Psychometrika 1965; 30: 179–85. doi:10.1007/BF02289447 http://www.ncbi.nlm.nih.gov/pubmed/14306381

²⁶ Auerswald M, Moshagen M. How to determine the number of factors to retain in exploratory factor analysis: a comparison of extraction methods under realistic conditions. Psychol Methods 2019; 24: 468–91. doi:10.1037/met0000200 http://www.ncbi.nlm.nih.gov/pubmed/30667242

²⁷ Cappelleri JC, Jason Lundy J, Hays RD. Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures. Clin Ther 2014; 36: 648–62. doi:10.1016/j.clinthera.2014.04.006 http://www.ncbi.nlm.nih.gov/pubmed/24811753

²⁸ Lt H, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modeling 1999; 6: 1–55. doi:10.1080/10705519909540118

²⁹ Streiner DL. Starting at the beginning: an introduction to coefficient alpha and internal consistency. J Pers Assess 2003; 80: 99–103. doi:10.1207/S15327752JPA8001_18 http://www.ncbi.nlm.nih.gov/pubmed/12584072

³⁰ Keni R, Alexander A, Nayak PG, et al. COVID-19: emergence, spread, possible treatments, and global burden. Front Public Health 2020; 8: 216. doi:10.3389/fpubh.2020.00216 http://www.ncbi.nlm.nih.gov/pubmed/32574299

³¹ Kantor J. Reliability and photographic equivalency of the scar cosmesis assessment and rating (scar) scale, an outcome measure for postoperative scars. JAMA Dermatol 2017; 153: 55–60. doi:10.1001/jamadermatol.2016.3757 http://www.ncbi.nlm.nih.gov/pubmed/27806156

³² Kantor J. The scar (scar cosmesis assessment and rating) scale: development and validation of a new outcome measure for postoperative scar assessment. Br J Dermatol 2016; 175: 1394–6. doi:10.1111/bjd.14812 http://www.ncbi.nlm.nih.gov/pubmed/27292082