Validity and reliability of the Serbian COVID Stress Scales

Abstract

This study aimed to generate a linguistic equivalent of the COVID Stress Scales (CSS) in the Serbian language and examine its psychometric characteristics. Data were collected from September to December 2020 among the general population of three cities in Republic of Serbia and Republic of Srpska, countries where the Serbian language is spoken. Participants completed a socio-demographic questionnaire, followed by the CSS and Perceived Stress Scale (PSS). The CSS was validated using the standard methodology (i.e., forward and backward translations, pilot testing). The reliability of the Serbian CSS was assessed using Cronbach’s alpha and McDonald’s omega coefficients and convergent validity was evaluated by correlating the CSS with PSS. Confirmatory factor analysis was performed to examine the construct validity of the Serbian CSS. This study included 961 persons (52.8% males and 47.2% females). The Cronbach’s alpha coefficient of the Serbian CSS was 0.964 and McDonald’s omega was 0.964. The Serbian CSS with 36 items and a six-factorial structure showed a measurement model with a satisfactory fit for our population (CMIN/DF = 4.391; GFI = 0.991; RMSEA = 0.025). The CSS total and all domain scores significantly positively correlated with PSS total score. The Serbian version of the CSS is a valid and reliable questionnaire that can be used in assessing COVID-19-related distress experienced by Serbian speaking people during the COVID-19 pandemic as well as future epidemics and pandemics.

About the Authors:

Marija Milic

Contributed equally to this work with: Marija Milic, Jelena Dotlic

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft

Affiliation: Department of Epidemiology, Faculty of Medicine, University of Pristina Temporarily Seated in Kosovska Mitrovica, Kosovska Mitrovica, Kosovo, Serbia

Jelena Dotlic

Contributed equally to this work with: Marija Milic, Jelena Dotlic

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft

Affiliations Clinic for Obstetrics and Gynecology, Clinical Center of Serbia, University of Belgrade, Belgrade, Serbia, Faculty of Medicine, University of Belgrade, Belgrade, Serbia

Geoffrey S. Rachor

Roles Conceptualization, Formal analysis, Investigation, Methodology, Validation, Writing – review & editing

Affiliation: Department of Psychology, University of Regina, Regina, Canada

Gordon J. G. Asmundson

Roles Conceptualization, Formal analysis, Investigation, Methodology, Validation, Writing – review & editing

Affiliation: Department of Psychology, University of Regina, Regina, Canada

Bojan Joksimovic

Roles Conceptualization, Formal analysis, Investigation, Methodology, Writing – review & editing

Affiliation: Faculty of Medicine, University of East Sarajevo, Foca, Republic of Srpska Bosnia and Herzegovina

Jasmina Stevanovic

Roles Conceptualization, Data curation, Investigation, Methodology, Writing – review & editing

Affiliation: Department of Epidemiology, Faculty of Medicine, University of Pristina Temporarily Seated in Kosovska Mitrovica, Kosovska Mitrovica, Kosovo, Serbia

Dragoslav Lazic

Roles Conceptualization, Data curation, Investigation, Methodology, Writing – review & editing

Affiliation: Department of Dentistry, Faculty of Medicine, University of Pristina Temporarily Seated in Kosovska Mitrovica, Kosovska Mitrovica, Kosovo, Serbia

Zorica Stanojevic Ristic

Roles Conceptualization, Formal analysis, Investigation, Methodology, Writing – review & editing

Affiliation: Department of Pharmacology, Faculty of Medicine, University of Pristina Temporarily Seated in Kosovska Mitrovica, Kosovska Mitrovica, Kosovo, Serbia

Jelena Subaric Filimonovic

Roles Conceptualization, Data curation, Investigation, Methodology, Writing – review & editing

Affiliation: Department of Epidemiology, Faculty of Medicine, University of Pristina Temporarily Seated in Kosovska Mitrovica, Kosovska Mitrovica, Kosovo, Serbia

ORCID logo https://orcid.org/0000-0002-2161-2394

Nikoleta Radenkovic

Roles Conceptualization, Data curation, Investigation, Methodology, Writing – review & editing

Affiliation: Department of Epidemiology, Faculty of Medicine, University of Pristina Temporarily Seated in Kosovska Mitrovica, Kosovska Mitrovica, Kosovo, Serbia

Milica Cakic

Roles Conceptualization, Data curation, Investigation, Methodology, Writing – review & editing

Affiliation: Clinic for Obstetrics and Gynecology, Clinical Center of Serbia, University of Belgrade, Belgrade, Serbia

Tatjana Gazibara

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Supervision, Writing – original draft

* E-mail: [email protected]

Affiliation: Institute of Epidemiology, Faculty of Medicine, University of Belgrade, Belgrade, Serbia

ORCID logo https://orcid.org/0000-0002-9621-8375

Introduction

The COVID-19 pandemic has impacted the psychological health of the global population for more than one year. The first case of COVID-19 in the Republic of Serbia was identified on March 6, 2020. Until the end of June 2021, more than 700,000 confirmed cases of COVID-19 were registered, and more than 7,000 people died due to COVID-19-related complications. The dynamic of the epidemic over the past 15 months in Serbia could be divided in distinctive three waves (March to end-of-May 2020; July to the end of August 2020; October 2020 to the mid-May 2021, when the largest number of people caught the novel coronavirus and died as a result) [1].

Pandemic has led to substantial disruptions in health care delivery [2, 3], daily routines (e.g., mandatory use of face masks) [4, 5], and mental well-being [6, 7]. While mental health burden was first recognized among health care workers, due to excessive workload, deterioration in psychological well-being has also been observed among the general populations [8–11].

Researchers have demonstrated that stress and anxiety are common during epidemics and pandemics [12, 13]. The present pandemic is different from previous pandemics in recent decades in that everyday functioning for the global population has extraordinarily changed. For example, in addition to mandatory use of facemasks, governments have implemented social distancing protocols and closures of non-essential businesses, and several workplaces have shifted to having employees work remotely [4, 5]. To accurately define mental health challenges that people encounter daily during the COVID-19 pandemic, particularly the stress associated with potential contagion as well as the economic burden and potential xenophobia due to the closing of borders and reduction of international travel, there is an increased need for valid instruments to measure COVID-19-related distress [6, 7].

Recent studies suggested that the effect of COVID-19 on mental health differed between countries that applied various strategies to prevent and control the pandemic [14, 16]. Although an increase in frequency of manifested psychological symptoms during the pandemic was observed worldwide, especially symptoms of depression and anxiety, the impact on mental health was stronger in countries where the epidemiological situation was worse [14]. For example, in the United Kingdom, where lockdown was came into force somewhat later than in other countries in the European Union and funding of preventive measures was ten times lower compared to Germany, a more severe direct impact of COVID-19 on health, financial situation and families. By contrast, people in Germany were less optimistic regarding the end of the epidemic, but more worried about their life [15]. Another study that compared people in Poland and China found that less strict demand for face masks use in Poland compared to China was associated with more intense anxiety, depression and stress, as well as physical symptoms related to the COVID-19 infection [16].

Over the first wave of the pandemic, both Republic of Serbia and Republic of Srpska (Bosnia and Herzegovina) entered a lockdown, although residents who were living abroad were allowed to enter the country. During lockdown, people aged 65 and above were not allowed to leave their homes and were allocated a time slot once a week to shop for groceries, while curfew for all citizens lasted throughout the entire weekends. Reopening was gradual along with the improvement of the epidemiological situation [17]. Over the following months and the epidemic waves, various preventive and control measures were in place (working/schooling from home, closure of public spaces and ban of celebrations and gatherings of more than 5 people, reduction of working hours of cafes and restaurants, mandatory use of face masks inside and outside, social distancing, etc) except lockdown. The borders remained open during second and third epidemic wave although incoming people were requested to present certificate of negative PCR coronavirus test [17]. In Serbia, a national psychological telephone support service during lockdown. One quarter of people who used this service was older than 70 years, 63.6% were women, and one third reported having symptoms of anxiety and feelings of tension during the COVID-19 epidemic [18].

The COVID Stress Scales (CSS) were designed at the beginning of the COVID-19 pandemic in English language [19]. The CSS measures various facets of COVID-19-related distress, including fears about the dangerousness of COVID-19 and of contamination, socio-economic concerns, xenophobic attitudes, traumatic stress, compulsive checking and reassurance seeking symptoms. Based on data from literature and remarks from experts in the field of anxiety, the CSS included rigorous psychometric testing which resulted in reduction of the initial pool of items. The final version of the CSS is compact but versatile, which may allow researchers and practitioners to further adapt the scale in future pandemics [19].

The CSS has already been translated into several languages (e.g., psychometric properties have been assessed in Persian language) and has contributed to a growing body of evidence regarding the impacts of COVID-19-related distress and COVID stress syndrome [20, 21]. Because of the similar circumstances and public health measures across multiple countries, use of the CSS in various cultures could provide more nuanced insights into stress stemming from the COVID-19 pandemic. The purpose of this study was to generate a linguistic equivalent of the CSS in the Serbian language and examine its psychometric characteristics.

Material and methods

Setting

This study was carried out in two regions/countries where Serbian language is spoken: Republic of Serbia and Republic of Srpska (Bosnia and Herzegovina). Data were collected over the course of four months (September–December) in 2020. The recruitment of study participants was carried out in four randomly selected cities, three in Republic of Serbia (Belgrade, Kragujevac and Kosovska Mitrovica) and one in Republic of Srpska—Bosnia and Herzegovina (Foca).

We printed the names of all 29 official administrative cities in Serbia and 8 in Republic of Srpska on separate slips of paper folded them for blinding and placed them in a in a non-transparent container. In table of random numbers, we hit number nine and consequently draw from the container every ninth slip of paper with city names. In Republic of Serbia there are 5 regions (Belgrade, Vojvodina Region, Sumadija and Western Serbia region, Southern and Eastern Serbia region and Kosovo and Metohija region) with 197 municipalities [22]. Rural and suburban regions are divided and clustered into municipalities according to population size and geographical localizations. On the other hand, towns and cities form municipalities based on the number of residents. Smaller towns and cities are considered as one municipality while larger cities are divided into different municipalities for easier administration. Therefore, in Serbia there are 17 municipalities in the Belgrade region, 45 in the region of Vojvodina, 53 in the region of Sumadija and Western Serbia, 53 in the region of Southern and Eastern Serbia 29 in the region of Kosovo and Metohija [22]. However, because some cities had just one and others multiple municipalities, to overcome the discrepancies in the population size, we decided to distribute the questionnaires in only one randomly chosen municipality per city. We printed the names of all municipalities on separate slips of paper, folded them for blinding and placed them in one non-transparent container per city after which we chose one municipality from every container.

Selection of study participants

All people who came to the chosen municipality office headquarters to engage in regular administrative business were invited to participate in the study. In Serbia and Republic of Srpska (Bosnia and Herzegovina), civil administration affairs are just starting to function online. The vast majority of people who require any administration related to taxation or are in need of official certificates (birth, death, marriage, divorce, etc) and other paperwork are required to come in person to the municipality office. Thus, many people visit their municipality office quite often. This approach enabled us to select people at random and include individuals of different socio-demographic backgrounds.

The inclusion criteria were: being ≥18 years, being of Serbian nationality and speaking Serbian language fluently. The exclusion criteria were: reporting psychiatric disorders previously diagnosed by a physician that could evidently negatively impact self-perception and comprehension (all F-coded diagnoses according to the International Classification of Diseases, 10th revision) and providing less than 90% of answers. In total, we approached 1,347 persons out of which approximately 71% fulfilled the study criteria. We excluded 19 individuals due to confirmed psychiatric and 367 due to not fulfilling 90% of the questionnaire. The final sample included approximately 0.36% of the population of the appraised cities.

The sample size is often dependent on the length of the questionnaire. Given that the CSS has 36 items, we aimed to minimize bias arising from the number of observations in order to perform a robust confirmatory factor analysis. Thus we opted for participant-to-item ratio of 25:1 (i.e. minimum sample size 36 x 25 = 900). Although some authors recommend that the participant-to-item ratio be at a minimum 5:1 [23], larger sample sizes could provide more meaningful factor loadings and factors and yield more generalizable results.

The Ethics Committee of the Faculty of Medicine, University of Pristina temporarily seated in Kosovska Mitrovica approved the study (approval no. 10-1285/1).

Instruments

All instruments applied in the study were self-administered in paper-and-pencil form. At any moment, at least one researcher was present to provide all necessary explanations. Socio-demographic characteristics, lifestyle and habits as well as short medical history data were gathered from all study participants using a general demographics questionnaire.

COVID Stress Scales (CSS). The CSS was constructed to better understand and assess COVID-19-related distress and health-related anxiety during times of pandemic [19]. The scale has 36 items which are grouped into six domains, including 1. danger (DAN), 2. socio-economic consequences (SEC), 3. xenophobia (XEN), 4. contamination (CON), 5. traumatic stress symptoms (TSS) and 6. compulsive checking and reassurance seeking (CHE). Items are rated on a 5-point scale ranging from 0 (not at all) to 4 (extremely) for fear-related items in the DAN, SEC, XEN, and CON domains. The TSS and CHE domains are also rated from 0 (never) to 4 (almost always), however, responses are related to frequency rather than intensity. The scores for each of the six domains are calculated as the sum of ratings for each item in that domain. Higher scores indicate more intense or more frequent perceptions. The composite CSS score (i.e., total score) is the sum of all six domain scores and ranges from 0 (i.e., low COVID-19-related distress) to 144 (i.e., severe COVID-19-related distress).

The Perceived Stress Scale (PSS) is the most frequently used scale to measure perceived stress [24]. The scale assesses the degree to which individuals evaluate their life circumstances and situations as stressful. The PSS includes not only items regarding the symptoms of psychological distress (e.g., anxiety and depression) but also encompasses perception of unpredictability, uncontrollability, and sense of overload, making it sensitive to both current life conditions and expectations for future. The PSS consisted of 10 items rated from 0 (never) to 4 (very often) [24]. The PSS scores are obtained by reversing responses to the positive items (i.e., items 4, 5, 7 and 8) and then adding the item-level scores across all items. Higher total scores indicate greater levels of stress. In the available literature norms for the original PSS are based on US population and encompass the mean +/- standard deviation values of PSS total score according to gender, age groups and race [24]. The PSS has previously been translated and validated in the Serbian population [25].

Translation of the COVID Stress Scales

The translation and use of the CSS to Serbian was approved by the authors of the scale. We first translated the CSS from English to Serbian (forward translation). The translations were performed by two independent translators who were fluent in English. The two forward translated versions were analyzed. A third professional English translator, who was blinded to the original version of the questionnaire, performed back translation from Serbian to English language. Afterwards, all translators discussed the three translations to identify discrepancies.

After some modifications and consensus between the translators (S1 File), we generated the final version of the CSS in Serbian (S1 File). This version was tested on 10 adults, for understanding and additional comments about clarity. No appreciable remarks were noted in this process. The final version was then emailed to the questionnaire authors for approval. Upon discussion about the Serbian translation with the authors of the CSS and minor adjustments, we were granted the permission to test the psychometric properties of the Serbian CSS. The Serbian translation (in both Cyrillic [official alphabet] and Latin alphabet) can be found on the official CSS website https://coronaphobia.org/professional-resources/.

Statistical analysis

We used mean value and standard deviation (SD) of demographics as well as CSS and PSS total scores to characterize the study sample. To describe the CSS we also analyzed minimal and maximal values, skewness, and kurtosis for each item. Data analyses were performed using the SPSS 20.0 statistical software package (SPSS Inc., Chicago, IL, U.S.A.) and the JASP version 0.14.1 (http://jasp-stat.org). AMOS software version 18.0 was used to conduct CFA.

The scale internal consistency and reliability was evaluated by examining Cronbach’s alpha and McDonald’s omega coefficients [25, 26]. The alpha coefficient is a common indicator of whether the scale is internally consistent. The omega coefficient represents composite reliability i.e. construct reliability and is therefore recommended to estimate “matter-of-fact” reliability of the scale, as the coefficient uses the strength of association between items and constructs as well as item-specific measurement errors. Adequate levels for both coefficients are >0.7 [27]. Discriminating characteristics of the scale items were examined by means of Corrected Item–Total Correlation (CI–TC). These values take into account the relationships of one item with the score of the remaining items in the scale. Items with CI–TC ≥0.40 are regarded as suitable as they are consistent with the averaged results of other items. Finally, we used Hotelling’s t-squared test to assess the existence of a meaningful difference between mean score values of all CSS items together and the hypothetical case in which items have equal scores [28, 29].

Initially, to define whether our data were suitable to run factor analysis, we observed Kaiser-Meyer-Olkin measure of sampling adequacy and Bartlett’s test of sphericity. Bearing in mind that the factorial structure of the CSS may differ between populations [19, 20], we decided to evaluate both the six and five-factor structure of the Serbian CSS using the confirmatory factor analysis (CFA). We examined several parameters on the CFA for appropriate fit. First, the χ2 test with degrees of freedom (df) allows reporting potential difference between the observed and the expected covariance matrices. Values of χ2/df (CMIN/DF) below 5.0 and a p-value of greater than 0.05 confirm the set (i.e., original model fit). However, evaluation of p-value in small and large samples is less reliable. Second, the root mean square error of approximation (RMSEA) avoids issues of sample size by analyzing the discrepancy between the hypothesized model with optimal estimates and the population covariance matrices. Adequate RMSEA value is below 0.08. Third, concurrence of hypothesized model with the observed covariance matrix is assessed by the goodness of fit index (GFI) and the adjusted goodness of fit index (AGFI). Fourth, the comparative fit index (CFI) shows the discrepancy between the data and the hypothesized model, while adjusting for the issues of sample size. Fifth, non-normed fit index (NNFI) is based on a comparison of the χ2 of the implied matrix with that of a null model in which all observed variables are uncorrelated. Values of the latter four indices above 0.9 indicate acceptable model fit [28, 29].

Criterion validity was tested using the Spearman’s correlation of the CSS domains and total scores with PSS total score. We also compared the CSS against PSS using the receiver operating characteristics (ROC) analysis to define potential cut-off scores of CSS according to PSS for higher levels of stress.

Results

Description of the study sample

A total of 961 participants were included in the analyses. The mean age of participants was 38.2 (SD = 14.1) years, of which 507 (52.8%) identified as males and 454 (47.2%) as females. The majority of participants reported having secondary education (54%), being permanently employed (50.8%), and having no chronic illnesses (59.8%). The largest proportion of participants were married (45.3%) and lived, on average, with 3.6 (SD = 1.6) household members. In our sample, 25% (n = 240) confirmed having contact with a COVID-19 positive person, but only 18.4% (n = 177) respondents were tested for COVID-19 due to typical symptoms, out of which 52 (3.9%) had COVID 19 at some point. To seek information about COVID-19, study participants most often used non-medical (58.9%) sources like media, internet and friends.

COVID Stress Scales scores

Average CSS item scores are presented in Table 1. Data were not normally distributed, but skewness and kurtosis were appropriate.

[Figure omitted. See PDF.]

Table 1. Average COVID Stress Scales item scores and their reliability parameters.

https://doi.org/10.1371/journal.pone.0259062.t001

In the Serbian population, scores ranged from 0 to 4 for all items; however, all items marked as four by participants occurred significantly less often compared to other ratings. The mean total CSS was 35.4 (SD = 25.9), with a range from 0 to 144, suggesting moderately high overall stress due to COVID-19. The highest average score per domain was achieved for DAN scale and the lowest for TSS scale (Table 2).

[Figure omitted. See PDF.]

Table 2. Average values, Cronbach’s alpha and correlation coefficients between domains/factors.

https://doi.org/10.1371/journal.pone.0259062.t002

Internal consistency

The Cronbach’s alpha coefficient of the Serbian CSS was 0.964 (CI 0.960 to 0.967) and McDonald’s omega was 0.964 (CI 0.961 to 0.967). When based on standardized items, the alpha coefficient was 0.965. We observed that Cronbach’s alpha was adequate for each CSS domain (Table 2). The values of Cronbach’s alpha coefficient if-item-deleted were appropriate for all items. All coefficients ranged from 0.962 to 0.964. However, the highest coefficients were observed when items 3, 33, and 36 were deleted (Table 1). Cronbach’s alpha values for the CSS domains were also all above 0.8 (Table 2).

Based on the Hotelling’s t-squared test, there was a significant difference (HT2 = 2255.146; F = 62.151; p = 0.001) between item scores. The highest average score was achieved for items 22 (CON) and 3 (DAN), while item 30 (TSS) had the lowest average score. The values of the CI-TC coefficient for the Serbian CSS were higher than 0.40 for all items, with the lowest of 0.463 for item #3 (DAN).

Construct validity

The sampling adequacy according to the KMO criteria was 0.954 and Bartlett’s test of sphericity showed the probability value of p<0.001, suggesting that data are suitable for factor analysis. The Serbian CSS with 36 items and a six-factor structure showed a measurement model with a satisfactory fit for our population (CMIN/DF = 4.391; p = 0.001; GFI = 0.991; AGFI = 0.847; CFI = 0.996; NNFI = 0.995; RMSEA = 0.025). On the other hand, a five-factor structure was somewhat less adequate (CMIN/DF = 5.881; p = 0.001; GFI = 0.708; AGFI = 0.799; CFI = 0.829; NNFI = 0.815; RMSEA = 0.097). Therefore, we opted to accept and further explore the six-factor Serbian CSS.

The observed low covariance between the six factors confirmed adequate construct validity of the CSS (Fig 1). The only covariance above 0.5 was observed for CON and XEN domains. Yet, covariance was observed between items 30, 35, 29 and 30; 20 and 21; 10 and 12; 7, 10 and 8; and 3 and 4, indicating that they were part of a similar construct. Nevertheless, as the overall model was adequate, we opted to keep all the original items in the Serbian CSS.

[Figure omitted. See PDF.]

Fig 1. Confirmatory factor analysis for COVID Stress Scales.

https://doi.org/10.1371/journal.pone.0259062.g001

Criterion validity

Total PSS scores in our sample were quite high, with a mean of 21.4 (SD = 5.25) and ranged of 7 to 46. Only 60 participants had stress levels below the highest average of norm groups (14 points) measured for the United States (23).

The CSS total and all domain scores significantly positively correlated with PSS total scores. Moreover, inter-correlations for CSS domains were all statistically significant indicating good convergent validity (Table 3).

[Figure omitted. See PDF.]

Table 3. Correlations between the COVID Stress Scales and Perceived Stress Scale total scores.

https://doi.org/10.1371/journal.pone.0259062.t003

Based on the ROC curve analysis, the stress levels measured by the CSS adequately explained 65.9% of cases compared with PSS (p = 0.001). The cut-off level of CSS above which the stress could be considered as high in our study was 24.5 score (specificity = 62.8%; sensitivity = 61.7%; Fig 2).

[Figure omitted. See PDF.]

Fig 2. ROC analysis of COVID Stress Scales according to Perceived Stress Scale.

https://doi.org/10.1371/journal.pone.0259062.g002

Discussion

Results of this study suggest that the CSS administered in Serbian language among Serbians has a 6-factorial structure, which mirrors the original construct of the questionnaire. There was no compelling metric evidence that any of the items should be omitted. The questionnaire showed adequate overall internal consistency, whereby all coefficients were >0.8, which demonstrates excellent internal consistency of the domains. The CSS is comparable to another well-established questionnaire examining stress, the PSS. Based on the ROC analysis, the CSS could potentially be used in rapid triage of persons who are at risk of higher stress levels requiring clinical attention.

Our study was conducted in a community sample of adults, as was done in the original validation study [19]. The psychometric properties of the CSS in Persian have also been examined in a sample of persons with anxiety and obsessive-compulsive disorders in Iran [20]. In both studies, the participants scored the highest on the DAN scale, which is consistent with our results [19, 20]. On the other hand, our participants scored the lowest on the TSS scale, while in Iran, for example, the lowest scores were observed on the SEC scale [20]. Yet, the selection of the examined population in Iran was considerably different from ours (i.e., patients from psychiatric hospitals and clinical centers vs. community sample) and could have, therefore, contributed to the observed differences.

The study sample was representative of the population in both countries as there were no significant differences in socio-demographic data of questionnaire respondents and the general populations of Republic of Serbia and Republic of Srpska. The mean age of our study participants was 38.2 +/- 14.1 years. In Serbia children up to 14 years encompass 14.3%, 65.0% of citizens are aged 15–65 years, while seniors over 65 years make 20.7% of the population. In Republic of Srpska the age group up to 14 years includes 14.1% of the population, 68.8% is aged 15–65, while 17.1% of the population is aged over 65 years [30].

In our study 52.8% participants were males and 47.2% females. Out of 6,945,235 inhabitants of the Republic of Serbia 48.7% are males and 51.3% are females, while out of 1,147,902 people who live in the Republic of Srpska 48.9% are males and 51.1% are females. In Serbia 73.1% of the population is married, while 57.3% of the population of Republic of Srpska is married. In our study the number of married participants was somewhat lower, but being married was still the most common relationship status (45.3%) [31].

The majority of our study participants reported having secondary education (54%) and being permanently employed (50.8%). About 2.9 million people in Serbia are employed and 292,000 are unemployed. As for Republic of Srpska about 350,670 people are employed and 118,189 are unemployed. In Serbia about 13.7% of the population has no or incomplete primary education, 20.8% have completed only primary school, 48.9% have completed secondary school, and 16.3% of people have higher or high i.e. university level of education. In Republic of Srpska about 15.2% of the population has no or incomplete primary education, 21.2% have completed only primary school, 50.6% have completed secondary school, and 12.8% of people have higher or high i.e. university level of education.

Results of our study could be additionally explained by the timing of our survey relative to the onset of the pandemic. Specifically, the index case in Serbia was identified in the first week of March 2020; yet, it is possible that the SARS-CoV2 was circulating in the population in early 2020 or late 2019 [32]. The current study was conducted during September to December of 2020, which may have allowed enough time for individuals to adjust to the circumstances of the pandemic, such as social distancing and working remotely, so that the experience was not perceived as traumatic at this point in time. If we had surveyed the population at the very beginning of the pandemic it is possible that the results could have been different. Nevertheless, the pandemic is still ongoing in Serbia. While the frequency of newly diagnosed people with COVID-19 increased in June and July and decreased in August and September, the highest incidence rates per 100,000 were observed in November and early December 2020. Consequently, the study period covered both low and high rates of COVID-19 enabling us to test their potentially different impact on stress level in our population.

Differences in the factor-structure of the CSS may reflect a specific context or purpose of the questionnaire. We found that 6-factor solution is suited for our community sample. Contrary, the original validation study, utilizing parallel analyses, demonstrated that a 5-factor solution was sufficiently stable for the Canadian and US community-based samples [19]. This solution was replicated in Iran among persons with anxiety and obsessive-compulsive disorders [20]. In this 5-factor model, DAN and CON subscales were merged and observed as a single construct [19, 20]. Our findings suggest that in the Serbian population, a distinction between perceptions of the pandemic as dangerous and disrupting everyday functioning on the one hand, and getting exposed to virus in the immediate environment on the other hand, should be made.

We observed that the CSS internal consistency as measured by both the alpha and omega coefficients was satisfactory. The original validation study used only the alpha coefficient to test the internal consistency of the scale. The validation of the CSS in the Iranian sample used both alpha and omega coefficients, which were similar to ours (≥0.88). It has been discussed that the omega coefficient is more reliable than the alpha coefficient as it accounts for the variability of the covariance [33]. For this reason, it is recommended that the omega coefficient is included in psychometric testing.

Previous validation studies observed good convergent validity of the CSS in English and Persian [19, 20]. We administered the PSS alongside the CSS to examine whether the Serbian CSS could distinguish the persons who experience more intense stress from those who do not. We observed that the level of sensitivity and specificity of the CSS in Serbian might be utilized as triage orientation to identify potential persons who need systematic mental health support. For this reason, further exploration of CSS in screening for COVID-19-related distress is warranted.

Strengths and limitations

The strength of this study is the fact that we selected a community-based sample (i.e., general population from several cities, including the capital of the Republic of Serbia) in two countries. Having the adequate participant-to-questionnaire item ratio the conditions for factor analysis were satisfied and bias arising from the number of observations was minimized [23]. Finally, the inclusion of the McDonald’s omega coefficient strengthened the interpretation of the scale internal consistency and reliability.

Some limitations need to be mentioned. The study sample might not have equal probability of selection, but is to an extent biased in favor of residents with reasons to go to municipality headquarters. At the time of pandemics a significant number of people might have been reluctant to go out if not absolutely necessary, as suggested by the medical authorities. Therefore, these individuals have not been included in our study. However, in Serbia and Bosnia and Herzegovina, many people visit municipal government offices to complete with different administrative procedures regardless of their gender, age, socio-economic status or political preferences. Moreover, we did not include a longitudinal follow-up of the study sample. As such, we were unable to re-test the selected population. In this way, the examination of stability of the CSS in Serbian language is missing. We did not include testing of any other parameter except the PSS, so this study is limited in terms of criterion validity. The inclusion of depression might have been useful to strengthen the convergent validity.

Conclusion

In conclusion, this study suggests that the CSS in the Serbian general population is valid and reliable questionnaire to assess distress related to the COVID-19 pandemic. The CSS in Serbian language incorporates six domains. We recommend this questionnaire in further research about mental health during the current pandemic as well as in research on the mental health impacts of future epidemics and pandemics.

Supporting information

S1 File. Additional description of the COVID Stress Scales translation and modification and adjustment of COVID Stress Scales to Serbian language.

https://doi.org/10.1371/journal.pone.0259062.s001

(DOC)

S1 Data.

https://doi.org/10.1371/journal.pone.0259062.s002

(XLS)

Acknowledgments

We are grateful to Ms. Bojana Milic, Bachelor of Laws, for help with administration and data entry. We are thankful to all the municipality administrations in Serbia and Republic of Srpska (Bosnia and Herzegovina) and all study participants who made this study possible.

Citation: Milic M, Dotlic J, Rachor GS, Asmundson GJG, Joksimovic B, Stevanovic J, et al. (2021) Validity and reliability of the Serbian COVID Stress Scales. PLoS ONE 16(10): e0259062. https://doi.org/10.1371/journal.pone.0259062

References

1. Institute of Public Health of Serbia "Dr Milan Jovanović Batut"Corona viral disease COVID-19. 2020 March 15 [Cited 2021 June 29]. Available from: http://www.batut.org.rs/download/aktuelno/230621.pdf

2. Bordea IR, Candrea S, Salagean T, Pop ID, Lucaciu O, Ilea A, et al. Impact of COVID-19 Pandemic on Healthcare Professionals and Oral Care Operational Services: A Systemic Review. Risk Manag Healthc Policy. 2021;14:453–63. pmid:33568961

3. Demeke HB, Merali S, Marks S, Pao LZ, Romero L, Sandhu P, et al. Trends in Use of Telehealth Among Health Centers During the COVID-19 Pandemic—United States, June 26-November 6, 2020. MMWR Morb Mortal Wkly Rep. 2021;70(7):240–4. pmid:33600385

4. Jacobs P, Ohinmaa AP. The enforcement of statewide mask wearing mandates to prevent COVID-19 in the US: an overview. F1000Res. 2020;9:1100. pmid:33520192

5. Tucho GT, Kumsa DM. Universal Use of Face Masks and Related Challenges During COVID-19 in Developing Countries. Risk Manag Healthc Policy. 2021;14:511–7. pmid:33603517

6. Fancourt D, Steptoe A, Bu F. Trajectories of anxiety and depressive symptoms during enforced isolation due to COVID-19 in England: a longitudinal observational study. Lancet Psychiatry. 2021;8(2):141–9. pmid:33308420

7. Hou WK, Lee TM, Liang L, Li TW, Liu H, Ettman CK, et al. Civil unrest, COVID-19 stressors, anxiety, and depression in the acute phase of the pandemic: a population-based study in Hong Kong.Soc Psychiatry Psychiatr Epidemiol. 2021 in press. https://doi.org/10.1007/s00127-021-02037-5 pmid:33590313

8. Dosil M, Ozamiz-Etxebarria N, Redondo I, Picaza M, Jaureguizar J. Psychological Symptoms in Health Professionals in Spain After the First Wave of the COVID-19 Pandemic. Front Psychol. 2020;11:606121. pmid:33391125

9. Brenner MH, Bhugra D. Front Psychiatry. Acceleration of Anxiety, Depression, and Suicide: Secondary Effects of Economic Disruption Related to COVID-19. 2020;11:592467.

10. Daly M, Sutin AR, Robinson E. Longitudinal changes in mental health and the COVID-19 pandemic: evidence from the UK Household Longitudinal Study. Psychol Med. 2020:1–10. pmid:33183370

11. Nkire N, Mrklas K, Hrabok M, Gusnowski A, Vuong W, Surood S, et al. COVID-19 Pandemic: Demographic Predictors of Self-Isolation or Self-Quarantine and Impact of Isolation and Quarantine on Perceived Stress, Anxiety, and Depression. Front Psychiatry. 2021;12:553468. pmid:33597900

12. Cheng SKW, Wong CW, Tsang J, Wong KC. Psychological distress and negative appraisals in survivors of severe acute respiratory syndrome (SARS) Psychol Med. 2004;34:1187–95. pmid:15697045

13. Rubin GJ, Amlôt R, Page L, Wessely S. Public perceptions, anxiety, and behaviour change in relation to the swine flu outbreak: cross sectional telephone survey. BMJ. 2009;339:b2651. pmid:19574308

14. Balmford B, Annan JD, Hargreaves JC, Altoe M, Bateman IJ. Cross-country comparisons of Covid-19: policy, politics and the price of life. Environ Resour Econ. 2020;76:1–27. pmid:32836862

15. Knolle F, Ronan l, Murray GK. The impact of the COVID-19 pandemic on mental health in the general population: a comparison between Germany and the UK. BMC Psychol. 2021;9(60):1–17. pmid:33892807

16. Wang C, Chudzicka-Czupała A, Grabowski D, Pan R, Adamus K, Wan X et al. The association between physical and mental health and face mask use during the COVID-19 pandemic: a comparison of two countries with different views and practices. Front Psychiatry. 2020;11:901. pmid:33033485

17. Republic of Serbia Ministry of Health. Institute for Public Health Reports. 2021 March 20 [Cited 2021 June 30]. Available from https://covid19.rs/eng-instituteforpublichealth-updates/

18. Stasevic-Karlicic I, Djordjevic V, Stasevic M, Subotic T, Filipovic Z, Ignjatovic-Ristic D et al. Perspectives on mental health services during the COVID-19 epidemic in Serbia. Srp Arh Celok Lek. 2020;148:379–82.

19. Taylor S, Landry CA, Paluszek MM, Fergus TS, McKay D, Asmundson GJG. Development and initial validation of the COVID Stress Scales Anxiety Disord. 2020;72:102232.

20. Khosravani V, Asmundson GJG, Taylor S, Sharifi Bastan F, Samimi Ardestani SM. The Persian COVID stress scales (Persian-CSS) and COVID-19-related stress reactions in patients with obsessive-compulsive and anxiety disorders. J Obsessive Compuls Relat Disord. 2021 in press. https://doi.org/10.1016/j.jocrd.2020.100615 pmid:33354499

21. Taylor S. COVID Stress Syndrome: Clinical and nosological considerations. Current Psychiatry Reports. 2021;23:19. https://doi.org/10.1007/s11920-021-01226-y pmid:33660068

22. Statistical Office of the Republic of Serbia. Demographic Yearbook. 2021 February 8 [Cited 2021 June 30]. Available at https://publikacije.stat.gov.rs/G2020/Pdf/G202014017.pdf

23. Comrey AL, Lee HB. A first course in factor analysis. 2nd Ed. Lawrence Erlbaum Associates, Inc;1992. pmid:11639968

24. Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. 1983;24(4):385–96. pmid:6668417

25. Jovanović Veljko & Gavrilov-jerkovic Vesna. More than a (negative) feeling: Validity of the Perceived Stress Scale in Serbian clinical and non-clinical samples. Psihologija. 2015; 48:5–18.

26. Bland JM, Altman DG. Cronbach’s alpha. Br Med J 1997;317:572 pmid:9055718

27. Dunn TJ, Baguley T, Brunsden V. From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. Br J Psychol. 2014;105:399–412. pmid:24844115

28. Kline RB. Principles and Practices of Structural Equation Modelling, 2nd Ed. New York, NY: The Guilford Press, 2005.

29. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate Data Analysis: A Global Perspective (7th Ed). 2010; New Jersey: Pearson Prentice Hall.

30. Republic Statistical Office. Statistical calendar of the Republic of Serbia. 2021 February 5 [Cited 2021 June 30]. Available from https://publikacije.stat.gov.rs/G2021/Pdf/G202117014.pdf

31. Institute of Statistics of the Republic of Srpska. Statistical Yearbook of Republic of Srpska. 2020 January 20 [Cited 2021 June 30]. Available from https://www.rzs.rs.ba/front/article/4582/?left_mi=None&add=None

32. Bogdanović M, Skadrić I, Atanasijević T, Stojković O, Popović V, Savić S et al. Case Report: Post-mortem Histopathological and Molecular Analyses of the Very First Documented COVID-19-Related Death in Europe. Front Med. 2021; 8:1–5. pmid:33681247

33. Crutzen R, Ygram Peters GJ. Scale quality: alpha is an inadequate estimate and factor-analytic evidence is needed first of all. Health Psychol Rev. 2017;11(3):242–247. pmid:26602990