1 INTRODUCTION
A new coronavirus disease that can cause fatal effects on humans, much like the Middle East respiratory syndrome (MERS) and severe acute respiratory syndrome (SARS), emerged in late 2019. The symptoms of the disease first became evident in China in December 2019, and the disease was diagnosed worldwide as Covid-19 at the beginning of the January 2020.1 According to the Covid-19 timeline drawn up by the World Health Organization (WHO) Covid-19, patients in China who were eventually diagnosed as having Covid-19 on December 31, 2019 showed symptoms similar to those of pneumonia. The disease was then reported to be a new coronavirus disease. The WHO established its Incident Management Support Team to address the spread and impacts of the new coronavirus disease on January 1, 2020. On January 13, 2020, it was reported that the first Covid-19 case outside of China occurred in Thailand on January 8, 2020. In addition, as per official statements, the spread of the coronavirus disease was attributed to one patient who traveled to Wuhan, China and returned to Thailand.2 The WHO's status report on January 30, 2020, revealed a total of 7.818 cases of Covid-19 worldwide, 82 of which were located in 18 countries outside China.3
According to the WHO's case report of January 31, 2020, Covid-19 had spread to an additional country, taking the total to 19 countries outside China.4 The country that reported these new Covid-19 cases was Italy. With the addition of Italy, the number of European countries that had reported Covid-19 cases rose to four. Reports of Covid-19 in Italy started with two people, who mentioned the city of Wuhan as being part of their travel histories. Marta et al. explained that these two cases, which were recorded in Rome, included Chinese tourists. In addition, it is known that these two tourists traveled from Wuhan to Milano on January 23, following which they visited Rome.5 The direct flights between Wuhan and northern Italy, the Chinese workers' occupancy in the textile and fashion industry in Milan, and the touristic travels made over the Christmas day show the significance of Italy in spreading the coronavirus in Europe.6 The first cases in Italy were intense in Lombardia area and moved aggressively to northern regions,7 and hence a general lockdown was enforced by the government on March 9, 20208 which helped to control the pandemic. However, as the nationwide travel restrictions were flexed during summer, with the flow of tourists, new daily infections were inflated in August 20209 in the southern regions where no cases were identified before. Hence, the relationship of regional tourism and daily Covid-19 cases in Italy deserves a deeper analysis in the regions where the tourist activity has effected the spread of virus to understand its effect which is the main motivation of the current study.
Covid-19 spread to many continents and countries within a very short time, including Asia, America, and Europe. This disease was eventually declared as a pandemic worldwide. According to the WHO's weekly epidemiological update report of February 23, 2021 (It covers the time period till February 21), 110.763.898 approved cases occurred worldwide, of which 2.457.026 new cases were diagnosed in the last 7 days. Moreover, a total of 2.445.331 deaths were attributed to the pandemic, of which 66.359 new deaths had occurred in the last 7 days. The report of the same date showed 2.795.796 total cases in Italy, of which 84.977 were diagnosed as new (i.e., occurring in the last 7 days). Moreover, 95.486 deaths had occurred in total in Italy due to Covid-19, 2.130 of them within the last 7 days.10
Several studies have been conducted to investigate how rapidly the virus has spread globally in a short period of time; analyze its effect on society and economy; and suggest possible plans for recovery. A high population concentration heightens the risk of a region in terms of the number of Covid-19 infections.11 Rader et al. have studied the peakedness of the epidemic and concluded that big cities are prone to higher rates of infection spread and higher total attack rates over time compared to less populated cities.12 People in high density regions tend to have more face-to-face interaction at a shorter distance, leading to increased rate of infection transmission. This is an expected result considering that the virus spreads through the respiratory tract. On the other hand, Hamidi et al.13 observed that no direct proportional relationship exists between the density and Covid-19 infection rate in metropolitan areas after controlling social distance, socio-economics and healthcare infrastructure. Similarly, Sun et al. studied the effect of the population density and geographic factors on the spread of Covid-19 during the lockdown period. They found a negative correlation between spatial location and the number of daily cumulative Covid-19 infections in a region.14 Using the spatial regression model, Sannigrahi et al. conducted a comparative study to explore population and socio-demographic factors affecting Covid-19 casualties. They found that in addition to population, poverty, and income are major challenges against regulating the overall rise in the number of casualties of Covid-19 in Europe.15 In the study of Perone that used data related to 20 regions and 107 provinces in Italy, case fatality rate was found to be negatively correlated with factors like health care efficiency, physician density, and mean temperature. They also found that the highest mortality risk was focused in the north while the lowest risk was associated with southern areas.16
Although big cities create high risk environments because of close interaction between people, they also offer the advantages of easy accessibility to healthcare services and policy decision outcomes during a pandemic.13 However, even though healthcare facilities may provide necessary services to the city's residents, the disproportionately high number of patients overburdens the healthcare system. Thus, under pandemic conditions, new policies, protocols, and recommendations are urgently needed for management of patients requiring urgent surgeries, while simultaneously reducing the risks of spreading the virus to the other patients and healthcare professionals.17-19 Kapoor et al.20 listed several novel digital applications and solutions to improve the healthcare systems. For instance, telehealth services, such as virtual chatbots and webbots can help communicate between physician and patient without face-to-face interaction. As a result, stable patients can be discharged to their homes from the hospital, thereby reducing infection risk and patient density in healthcare facilities.
As areas with high population density are likely to receive visitors neighboring areas, they are prone to much higher cross-boundary infection risk.13 Kubota et al. conducted linear regression modeling and reported that cross-border human mobility is one of the reasons for the accelerated spread of Covid-19.21 Given this preliminary insight, many governments have decided to close their borders or impose serious restrictions during the pandemic. Wells et. al. investigated the impact of international travel and border control at the beginning of the pandemic in China using Monte Carlo simulations.22 Their results provide an idea of the effect of the restrictions on decreasing the case rate during the early stages of the pandemic. However, border lock-downs and mobility restrictions failed to prevent the virus from spreading globally, eventually wreaking severe negative effects on tourism and the economy.23 The tourism sector, which contributes considerably to countries' economies, has been the most negatively affected by this pandemic.24, 25 Effect of tourism on the spread of the virus has been investigated by many researchers. In the study of Roccetti and Casini, the relationship between Covid-19 spread in Italian regions and summer tourism 2020 was investigated.26 The study concluded that the increase in the number of daily cases experienced by all regions at the beginning of September 2020 was linked to summer holidays in Italy in August.7 A difference was found between certain regions when comparing the number of tourists arrived and the number of new infections. Similarly, Casini and Roccetti7 have explored the number of daily cases in different regions of Italy between middle of August and September 2020. They have built mathematical models to predict the total number of new Covid-19 cases by considering the number regional domestic tourists for each month where other factors such as population density and healthcare expenditure per region were also considered. Among the three prediction models built, namely linear, negative binomial regression, and cognitive models, the most promising result was obtained from the artificial neural networks model. Farzanegan et al. investigated the death rates in the countries considering the interaction between international tourism and Covid-19 cases. Using data from more than 90 countries and applying multiple regression analysis, a positive and significant correlation was found between the historical records of international tourism and the current cumulatively verified number of cases and deaths from Covid-19. The findings suggest greater control over the health aspects of the tourism industry and accountability of the main players of this industry according to health and safety standards.27 Gössling et al. have studied the effect of tourist flow from China and have associated the economical impact of the virus spread with the former pandemics.24 In this context, one of the goals of the current study is to investigate the impact of human mobility, in the form of domestic and foreign tourism on the spread of Covid-19 virus in Italy, where the highest number of cases had occurred after Chine at the beginning of the pandemic.
Data visualization is a powerful tool for exploring findings in data analysis.28-30 It basically provides a potential to unveil the important points and remarkable patterns of the dataset. Moreover, it may assist researchers and policymakers to discuss the statistical results on their analysis.31 Comba presented several example figures, graphs, and visualization techniques to understand the different effects of Covid-19 and predicted changes.32 Kodge reviewed the current status of Covid-19 infected cases in Maharashtra state in India using geographic visualization techniques.33 Galván-Tejada et al. proposed online visualization tools that allow an analysis of demographic data of the Mexican population. The purpose of these tools is to provide fast and up-to-date graphical information based on a data set directly from the National Government's Health Secretary, which is updated daily with information on SARS-CoV-2.30 In the relevant literature, several predictive algorithms34-36 and statistical models37, 38 were applied to analyze Covid-19 related data to estimate effect of various parameters related to Covid-19 infections. These studies considered predictive factors such as disease spread rate, infection risk, climate effects using deep learning-based algorithms.
Within this context, the current study applies data visualization techniques to examine the correlation between population density and human mobility, and their effect of healthcare services in terms of spread of Covid-19 in the dataset for Italy.39 Factors of population density and tourism intensity are investigated using ANOVA (analysis of variance) and Duncan analyses. Additionally, bidirectional long short-term memory (Bi-LSTM) network and autoregressive integrated moving average (ARIMA) algorithms are applied to build a forecasting model for the total Covid-19 cases in top five regions with the most Covid-19 cases in Italy.
2 MATERIALS AND METHODS
2.1 Data sets
2.1.1 Italy Covid-19 data set
The Covid-19 data set used in this study is licensed by the Italian Ministry of Civil Defense.39 It is updated every day at 18:30 after the press release. Italy Covid-19 data set files are shared on the GitHub page for information purposes.40 In the article, dpc-covid19-ita-province.csv and dpc-covid19-ita-regioni.csv files in this data set are used for visual data analysis (between 24 February 2020 and 24 February 2021). Table 1 represents semantic information of provinces and regions of Italy that is used in this article. The feature names in the data sets were translated into English. The region and province names, originally kept in Italian in order to avoid confusion.
TABLE 1. Feature names of Italy Covid-19 data sets according to provinces and regions| Feature names in the provinces data set | Feature names in the regions data set |
| date | date |
| region_name | region_name |
| province_name | lat |
| lat | long |
| long | hospitalized_with_symptoms |
| total_cases | intensive_care |
| total_hospitalized | |
| home_isolation | |
| total_positives | |
| total_change_positive | |
| new_positives | |
| discharged_healed | |
| deceased | |
| total_cases |
2.1.2 Italy resident population data set
Italy population data set used in this article was taken from the resident population distribution data provided by the Italian National Statistical Institute. The settled population distribution is considered to create two datasets according to all provinces and regions in Italy as of January 1, 2019.41 Table 2 represents semantic information of provinces and regions based on the distribution of the population by province and region respectively.
TABLE 2. Feature names of data sets containing Italy resident population by cities and regions| Feature names in the province population data set | Feature names in region population data set |
| province_name | region_name |
| total_male_population | total_male_population |
| total_female_population | total_female_population |
| total_population | total_population |
2.1.3 Italy tourist accommodation data set
Italy tourist accommodation data set was collected from the Italian National Institute of Statistics (Istat) Data Warehouse for this study. Information of this data set was taken from Occupancy in collective tourist accommodation—yearly data (residents by region) data table which is available in the Istat Data Warehouse that was selected for only year 2019 which contains total collective accommodation establishments (included both arrivals and night spent).41 The selected data set copied to an Excel file and then was converted to a csv file. Table 3 represents detail information about the number of tourist accommodation by regions.
TABLE 3. Feature names of data sets containing Italy tourist accommodation by regions| Feature names in the region tourist accommodation data set |
| region_name |
| numOfTourists_byRegions_2019 |
2.2 Data preprocessing
Python programming language is used for data analysis, data visualization and forecasting models in this study. Python is an effective and powerful programming language with data analysis and visualization tools. Pandas library is one of the important data analysis libraries available in Python for data processing. Both Italy Covid-19 data sets and Italy Resident Population data sets have been considered for data preprocessing. The datasets were cleaned by removing the cases with missing location information. After the data preprocessing was carried out, there are 107 provinces and 20 regions available in the dataset. In addition, latitude, and longitude information from the Covid-19 dataset were used for data visualization of population information. Twenty regions are considered which are Piemonte, Valle d'Aosta, Liguria, and Lombardia regions in the north west, Trentino-Alto Adige/Südtirol, Veneto, Friuli-Venezia Giulia and Emilia-Romagna regions in the north east, Toscana, Umbria, Marche, and Lazio regions in the center, Abruzzo, Molise, Campania, Puglia, Basilicata, and Calabria regions are located in the south, Sicilia and Sardegna regions which are located on the islands. Normally the P. A. Trento and P. A. Bolzano regions are subregions of the Trentino-Alto Adige/Südtirol region. To eliminate this complexity of meaning, the regional names of Trento and Bolzano are edited as Trentino Alto Adige/Südtirol in each data set.
In the data set shown in Table 3 regarding the tourist accommodation, no data preprocessing was performed regarding the region names. Only latitude and longitude information of Covid-19 data set was taken for data visualization of tourist accommodation information dataset.
2.3 Methods for analysis
In this part, we introduce a brief theory of the Bi-LSTM and ARIMA algorithms and explain their structures and forecasting model on Covid-19 cases in details.
2.3.1 Bi-LSTM
Bi-LSTM algorithm is designed based on the LSTM algorithm consisting of an exclusive Recurrent Neural Network architecture with memory and gates.42 Its memory structure has self-associated memory unit cells representing the transient state of the network at each time step. Each LSTM memory unit cell consists of an input gate, output gate and forget gate that is used to predict time series data from the previous information. In our case, the number of Covid-19 cases are fluctuating over time; the past and future information is necessary to obtain accurate forecasting models. Therefore, we use Bi-LSTM method and the overview of the algorithm is shown in Figure 1. As seen from the figure, Bi-LSTM algorithm includes backward and forward layers represented as in horizontal line and input-output layer countenancing forward flow information in directional line. S represents each LSTM layer containing hidden state which has memory cells represented as43 (1) (2) (3) [Image Omitted. See PDF.]where is the forward, is the backward hidden layer, is the weight at time t and and the values of memory unit at the previous time step (t-1) and future time step (t+1). Bi-LSTM provides better predictions than LSTM due to its bidirectional structure.
We train the Bi-LSTM network by minimizing the mean squared error between Covid-19 daily cases and predicted values. We implement all algorithms in the Google Colab Laboratory environment and use Keras Library in Python Programming Languages. The parameters of the Bi-LSTM algorithm are shown in Tables 4 and 5.
TABLE 4. Parameters and values used for the ARIMA and Bi-LSTM models| Method | Parameters | Values |
| Bi-LSTM | Learning rate | 0.001 |
| Optimizer | Adam | |
| Epochs | 100 | |
| Dropout | 0.2 | |
| ARIMA | (p,d,q) | (3,1,0) |
| Layer | Output shape | Number of parameter |
| Bidirectional | (None, 1, 256) | 133120 |
| Dropout | (None, 1, 256) | 0 |
| Bidirectional | (None, 128) | 164352 |
| Dropout | (None, 128) | 0 |
| Dense | (None, 1) | 129 |
2.3.2 ARIMA
There are different models created by using different methods to make estimates with the help of time series. Among these models, ARIMA models are the most known and widely used. ARIMA models, which assume a linear relationship between the data constituting the series and can model this linear relationship, can be successfully applied to time series that are stationary or stabilized by various statistical methods.
The ARIMA model is a model used for univariate time series forecasting. A time series without random white noise and exhibiting some kind of pattern can be modeled with ARIMA. Considering that the current time series data is a function of the past values, this method is widely preferred for future estimations as it does not have any restrictive assumptions. In addition, this method is used successfully in univariate short-term estimations and the most important feature is that the observed data should be a discrete and stationary series.44
The ARIMA Model utilises time series and delayed estimation errors that provides to predict next states according to the past values. An ARIMA model has three properties: p, d, and q. Where p is the order of the AR model, d is the degree of differentiation, and q is the order of the MA model.45
The equation of the ARIMA (p.d.q) model is shown in Equation 4. In Equation 4, is the mean of the series and and are vectors of parameters to characterize the series.46 (4)The model consists of four stages as seen in Figure 2. In Model identification, the most suitable (p, d, q) structure in the ARIMA model is selected. At this stage, it is the stage of determining whether the series is stationary or not, and when the series is stationary, which autoregressive process (AR(p)) and moving average process (MA(q)) models will be mixed. In the parameter estimation section, parameter estimations of the models are made. Diagnostic Checking takes place in two stages. First, the sample autocorrelation function of the original series is compared with the function of the series created by the model. If the two autocorrelation functions being compared seem too different from each other, the remodeling phase is started. If the difference between the two autocorrelations is small, the errors of the model are corrected. In the second stage, the compatibility of the model and residual analyses are generally checked for the adequacy of the model. In the forecasting phase, the aim is to make estimations close to the actual values that the variables take or will take in a certain period. Table 4 shows the parameters and values used for the ARIMA and Bi-LSTM models.
[Image Omitted. See PDF.]Relation between parameters of total cases, number of tourists and total population were explored through correlation analysis where Pearson correlation coefficient is employed. Pearson correlation coefficient values closer to 1 or were considered as strong correlation between the parameters.
Furthermore, for the statistical analysis between independent variables, ANOVA and Duncan tests were applied. ANOVA is used to analyze how the independent variables interact with each other and the effect of this interaction on the dependent variable. Additionally, Duncan test was used to compare the mean of each process with other means and individual test values in multiple comparisons
3 RESULTS
The visual data analysis of Covid-19 cases and resident population by regions and provinces were performed for 10 regions for region-based data visualizations such that five regions with the most cases and five regions with the least cases. Additionally, 20 provinces were examined for province-based data visualizations, 10 provinces with the most cases and 10 provinces with the least cases. Similarly, visual data analysis of nights spent at tourist accommodation establishments by regions were consisted of 10 regions for region-based data visualizations, five regions with most and least arrival and nights spent at tourist accommodation establishments by regions. In this way, the distribution of the tourist accommodation according to regions was investigated.
According to the population information of January 2019, Italy's total population is 60.359.546. Lombardia and Veneto are among the regions with the highest population, where the most Covid-19 cases occurred that have the highest tourist flow. On the contrary, Molise, Valle d'Aosta and Basilicata are among the regions with the least population, where the least Covid-19 cases occurred that have the least tourist flow. Lombardia and Veneto regions are located in the north of Italy, while Molise and Basilicata regions are located in the south of Italy. Although Valle d'Aosta is located in the north of Italy, it has a very small population and fewer Covid-19 cases occurred because it is located at the very edge.
The majority of Covid-19 cases occurring in Italy are located in the north of Italy (Figure 3). In direct proportion to this, the distribution of recovered or death cases occurred in the north.
[Image Omitted. See PDF.]The results of the visualization analysis, statistical methods and forecasting models are explained in respective sections below. The detailed results of the visualization and statistical analysis are provided as Supplementary Material.
3.1 Italy—Visual data analysis of Covid-19 cases by regions and provinces
Based on the data visualization in Figure 4A, the first five regions with the most Covid-19 cases among the 20 regions in Italy are clustered in the north of Italy. The total number of cases (1.666.376) in these five regions covers 58.50% of the total cases (2.848.564). Figure 4B shows that the majority of the top five regions, where the number of the Covid-19 case is the least, is located in the southern part of Italy. The total number of cases (111.462) in these five regions covers 3.91% of the total cases (2.848.564) in Italy. All the remaining provinces are located in the north, except Rome (Figure 4C). It can be stated that the reason for Rome to be affected mostly from Covid-19 is due to the high population density and the high tourist flow. It is known that the first two cases occurred in Italy were recorded in Rome. According to these records, it was stated that those cases were tourists and the city of Wuhan was located in their travel history. It is also known that these tourists traveled to Milano before coming to Rome and they came to Milano from Wuhan.5
[Image Omitted. See PDF.]In Figure 4D, it is clearly shown that the 10 provinces with the least number of cases are observed in the south of Italy and on the islands. It is understood from Figure 4A–D that the cases from the central regions of Italy to the north are more than the cases toward the south.
3.2 Italy—Visual data analysis of resident population by regions and provinces
Figure 4E shows that five regions with the highest population are distributed in the southern, central, and northern parts of Italy. Based on the analyses shown in Figure 4A–E, the fact that any region that has a border to other European countries causes the virus to spread. The number of resident populations in these regions are visualized in Figure 4F. Considering Figure 4F, it is possible to see that the five regions with the least population are positioned in the southern, central, and northern parts of Italy. The number of resident populations in these provinces are depicted in Figure 4G. It can be seen that the 10 provinces with the highest population are located in the southern, central, and northern parts of Italy. In addition, four other provinces except Rome are located in the north of Italy and close to each other. Although the population of the city of Rome is more than the population of Milano, the number of Covid-19 cases (201.581) in Milano is much more than the number of cases (162.576) in Rome. The number of cases (427.269) occurring in Milano, Torino, Brescia, and Bergamo provinces in the north of Italy constitutes 15% of the total number of cases (2.848.564). The remaining 103 provinces cover 85% of the total case. These four provinces appear to play a big role in the spread of the virus. The number of resident populations in these provinces are depicted in Figure 4H. Considering Figure 4H, it is possible to see that 10 provinces with the least population are placed in the southern, central, and northern parts of Italy. It is understood from Figure 4H that these provinces are located in the south or on the island. In addition, Isernia, Vibo Valentia, Aosta, and Oristano provinces in Figure 4H belong to the Molise, Calabria, Valle d'Aosta, and Sardegna regions, respectively. These regions are known as the places with the least Covid-19 cases in Figure 4B, respectively.
3.3 Italy—Visual data analysis of nights spent at tourist accommodation establishments by regions
According to the visualizations in Figure 4I, the majority of the top five regions is located in the northern part of Italy. Lazio is the only region located in the middle part of Italy in this data visualization.
Considering Figure 4I, the top five regions with the number of the most arrival and night spent at tourist accommodation in Italy are Lombardia, Lazio, Veneto, Emilia-Romagna, and Campania. The number of the most arrival and night spent at tourist accommodations in these regions are depicted in Figure 4I which shows the relation between tourist flow and Covid-19 cases. The number of the most arrival and night spent at tourist accommodation is shown in Figure 4J which shows that the five regions in Figure 4J are located in the southern, central, and northern parts of Italy.
Regions of Molise, Basilicata, and Valle d'Aosta in Figure 4J, exist in Figure 4B which shows top five regions with the least Covid-19 cases in Italy. In addition, regions of Molise, Basilicata, and Valle d'Aosta in Figure 4J, exist in Figure 4F which shows the top five regions with the least population in Italy depicting a relation between tourist flow, distribution of resident population and distribution of the Covid-19 cases.
3.4 Statistical analysis
Pearson correlation between total cases, number of tourists and total population revealed that there is a strong correlation between the parameters in question (Figure 5).
[Image Omitted. See PDF.]ANOVA is used to analyze how the independent variables interact with each other and the effect of this interaction on the dependent variable. ANOVA test analysis results are provided in Supplementary Materials. ANOVA gives the information whether the groups are different from each other or not. There is a very significant difference between the locations as in all parameters except for the total_change_positive.
For the ANOVA test, as a result is found to be significant (), additional analysis could be applied to understand between which groups the differences exist. Duncan test was used for that purpose. Duncan test results are provided in Supplementary Materials. According to the hospitalized_with_symptoms and total_hospitalized parameters, there is no difference for Campania and Veneto regions. Considering the intensive_care parameter, there is no difference in Veneto, Emilia-Romagna, and Piemonte regions. Similarly, for home_isolation and total_positives parameters, there is no difference in Emilia-Romagna and Veneto regions. According to total_change_positive, there is no difference between all regions. Considering new_positives factor, there is no difference in Piemonte, Emilia-Romagna, and Campania regions. For the discharged_healed parameter, there are no differences in Campania and Emilia-Romagna regions and Emilia-Romagna, Piemonte, and Veneto regions. Considering deceased parameter, there is no difference in Piemonte and Emilia-Romagna regions.
Additionally, analyses of correlation and descriptive statistics are provided in Supplementary Materials. We observed from the results table that there is a high correlation between hospitalized_with_symptoms and intensive_care parameters.
3.5 Forecasting models for Covid-19 cases
As a result of the analysis, it is seen that the top five regions (Lombardia, Veneto, Campania, Emilia-Romagna, and Piemonte) with the most Covid-19 cases cover 58.50% of Covid-19 cases occurring in Italy. Therefore, estimating the total number of confirmed cases, healing, and deaths occurring in these regions plays an important role in preventing the spread of Covid-19 cases. That is why, an estimation model for confirmed daily cases, healed, and death numbers occurring in these five regions, is created using Bi-LSTM (Figure 1) and ARIMA algorithms.
In these five regions, 367 days of Covid-19 case data (between 24 February 2020 and 24 February 2021) was used for estimation with the Bi-LSTM and ARIMA methods. After the data was normalized, it was split into train and test sets. In this article, timestamp was selected as 3, which means value of 3 days before was used to predict next value because the case values change with unexpected difference every day. Two hundred ninety-three days were taken to train the model, and remained last 60 days were used for testing.
A 2-layer Bidirectional LSTM was added and dropout was used for each layer. Thus, excessive memorization of the model was prevented. Model summary is shown in Table 5. There are 297.601 total and trainable parameters. The model in Table 5 is used to forecast Covid-19 cases of these five regions. Forecasting performed for Lombardia, Veneto, Campania, Emilia-Romagna, and Piemonte regions are shown in Figure 4A.
According to Table 6, it observed that ARIMA algorithm outperforms Bi-LSTM algorithm in terms of RMSE values. In the future studies, ARIMA algorithm could be a good candidate to consider for forecasting Covid-19 data (Figure 6).
[Image Omitted. See PDF.] TABLE 6. Comparison of ARIMA and Bi-LSTM forecasting algorithms for top five regions with the most Covid-19 cases in Italy| Regions | RMSE for ARIMA | RMSE for Bi-LSTM |
| Lombardia | 766.79 | 876.02 |
| Veneto | 633.02 | 650.30 |
| Campania | 330.76 | 344.82 |
| Emilia-Romagna | 270.00 | 340.57 |
| Piemonte | 291.82 | 372.96 |
4 DISCUSSION
The analysis reveals that the top five regions with the highest number of Covid-19 cases are located in the north of Italy. It could be concluded that these regions interact with each other in the proliferation of the epidemic. Lombardia and Veneto are among the top five regions with the most Covid-19 cases, which are the most crowded regions in Italy as well. Similarly, Basilicata, Molise, and Valle d'Aosta which are listed in the last five regions with the least number of cases are among the least populated regions in Italy. Thus, it can be stated that population of the regions are related to Covid-19 spread and Healthcare services needs to be canalized to highly populated areas. Until the spread of the virus is under control, transitions to these areas needs to be halted. Moreover, as the management of healthcare services is critical and moreover challenging in crowded areas, local healthcare facilities could be utilized to prevent secondary transmissions by reducing unnecessary hospitalizations. Because of congested hospitals, postponed treatments of chronic diseases and late operations, epidemic posed serious challenges to healthcare organizations.47 Hence taking in consideration the population, tourist flow and regional interactions within the country, restructuring the healthcare services, that is, care setting, personnel requirements, and equipment needs, could mitigate the load of the healthcare systems.48 As Covid-19 outbreak posed indirect challenges for the diagnosis and treatment of other diseases and urgent operations, alternative care models and policies such as transitional care, could be planned and utilized in healthcare management.
Although the population of some regions is high (e.g., Lazio and Campania regions), it is seen that there are fewer cases since there are no other regions in the near border where Covid-19 cases occurred intensely. One of the biggest reasons why more cases occurred in the northern regions of Italy is that it is closer to other countries. Although the populations of the regions that do not have border with other countries or regions are high, there have been less cases in the islands and southern regions (e.g., Sardegna, Calabria regions). Similarly, location of a region is observed in analysis for Rome and Milano. Even though Rome is more crowded than Milano, Covid-19 cases were lower in Rome. Because of its central location that is isolated from other European countries, and far from Milano, it is isolated because of its location compared to Milano. Considering the northern regions, although Valle d'Aosta is located in the north of Italy, it has a very small population and fewer Covid-19 cases occurred because it is location at the very edge. Therefore, depending on the population, geographical location also played an important role in the occurrence of the cases. Healthcare service planning should consider not only population, but also the location of the regions with respect to their border with other countries.
With respect to the tourism factor and the total Covid-19 case, there are common regions in the first and the last five regions of population and of Covid-19 data sets. These regions are Lombardia, Emilia-Romagna and Veneto. This shows that the tourist flow to the regions is related to Covid-19 spread. All of the mentioned regions are located in the north of Italy. The number of tourists flows in these three regions (112.289.566) covers 39.75% of the total tourist flow (282.448.020) in the country. In addition, the number of Covid-19 cases (1.166.373) occurring in these 3 regions constitutes 41% of the total number of Covid-19 cases (2.848.564). Therefore, the tourism factor has an important role in the spread of Covid-19 cases. Hence in the pandemic conditions, cross border movements should be ceased immediately. Thus, by temporarily stopping the flow, it will be possible to use healthcare services in crowded areas more effectively. For touristic regions, the traffic should be monitored on a monthly basis which would be modeled by machine learning algorithms and necessary measures could be taken beforehand so that emergency aid can be dispatched quickly at peak times when pandemic situations occur. To control the spread of the pandemic, utilization of high-tech solutions could provide effective defense against epidemic. Accordingly, advanced systems such as big data analysis, artificial intelligence and cloud computing were employed by China to control the spread.49 In order to reduce the negative outcomes of pandemics on the healthcare system and economy, integration of such intelligent technologies into the healthcare system is inevitable by controlling the spread quickly. As new occurrences of viral infections are expected in the near future, governments need to reconsider their emergency plans by incorporating innovative systems in handling pandemic conditions. Supportively, during pandemic periods, forecasting methods may provide promising solutions for short term estimations of Covid-19 cases. Hence, based on the estimations, healthcare system organizations may take necessary measures so as to respond to the increasing demand of the diagnosis and treatment of the infection.
In the study of Roccetti and Casini, the relationship between Covid-19 spread in Italian regions and summer tourism 2020 was investigated with the generalized linear model (GLM).26 In this study, we examined the impact of population and tourism factors on the occurrence of Covid-19 cases in Italy through data analysis and visualization. Population density and tourism density factors were investigated using ANOVA and Duncan analyses. In addition, Bi-LSTM and ARIMA algorithms are applied to create a forecasting model for total Covid-19 cases in the top five regions with the highest number of Covid-19 cases in Italy. The findings suggest that the spread of Covid-19 cases is influenced by the number of resident populations depending on the geographic location of the regions. In this article, our main goal is to investigate the impact of Covid-19 cases based on population size and tourism factors in certain regions of Italy with visual data analysis. Kaim et al. investigated the impact of Covid-19 in Israel through a tool.50 Study of Mirri et al. explored the potential relationship between the daily distribution of PM2.5 air pollutants and the initial spread of Covid-19 in New York.51 However, in our study, the effect of population and tourism factors on the number of Covid-19 cases occurred in Italy was investigated using real data. Casini and Roccetti have examined the growth rate of daily SARS-CoV-2 cases in all regions of Italy and shown a link between the reopening of schools and the resurgence of the virus in Italy.7 Nevertheless, in the current study, the effect of population and tourism factors on the number of Covid-19 cases in Italy has been shown. Table 7 shows the comparison of the studies conducted with our study.
TABLE 7. Comparison of current models based on Covid-19 data| Author | Dataset | Country | Methods | Covid-19 data | RMSE |
| Ceylan et al.52 | 45 days | France, Italy, Spain | ARIMA | Total confirmed cases | For Italy ARIMA(3,2,1): 1654.6600 |
| Punn et al.53 | 71 days | WorldWide | Support vector regression deep neural network long short-term memory polynomial regression | Total confirmed cases | LSTM: 15647.64 |
| Moftakhar et al.54 | 71 days | Iran | Artificial neural networks ARIMA | Daily confirmed cases | ARIMA: 1539.43 |
| Chimmula and Zhang55 | Until March 31, 2020. | Canada | LSTM | Daily Confirmed cases | 34.83 |
| Kıbaş et al.56 | For United Kingdom: 94 days | Various European countries | ARIMA, NARNN, LSTM | Daily confirmed cases | For United Kingdom ARIMA: 2405.1984 NARNN: 9991.4094 LSTM: 5482.2361 |
| Shahid et al57 | 110 days | WorldWide | ARIMA, SVR, LSTM, Bİ-LSTM | Daily Confirmed Cases | For Italy: ARIMA: 3612.8165 |
| Tian et al.58 | 84 days | South Korea, Italy, United State, Taiwan, Japan, Germany | Hierarchical bayes model (HBM), LSTM, hidden Markov model (HMM) | Daily confirmed cases | For Italy: HBM: 226.22 LSTM: 258.65 HMM:436.86 |
| Our study | 367 days | Italy | ARIMA, Bi-LSTM | Daily confirmed cases | ARIMA: 270.00 Bi-LSTM: 340.57 |
One of the results of the analysis reveals strong correlations between the number of patients in intensive care units (intensive_care) and the patients hospitalized with symptoms (hospitalized_with_symptoms) parameters and between number of patients in intensive care unit (intensive_care) and patients hospitalized (total_hospitalized) parameters. It is mentioned that nearly one third of the Covid-19 infected patients are admitted to intensive care units (ICU) at the beginning of the pandemic.59 Intensive care units are critical services in terms of treatment and also spread of the virus. Hence several essential requirements for ICU units need to be considered which could enhance control of the infection. Necessary measures must be taken to prevent the transmission of the virus among healthcare staff. Moreover, nonmedical staff involved in the admission of the patients to ICU needs to be trained on various workflows such as isolation and prevention of the virus transmission. Reorganization of ICU processes for quick diagnosis and segregation, organization, and infection inhibition could be crucial for not only healthcare personnel and Covid-19 infected patients but also patients with other diseases to reduce the risks of transmission.
5 CONCLUSION
In this article, we examined the population and tourism factors' effect on the realization of Covid-19 cases in Italy through data analysis and visualization. The findings show that the spread of Covid-19 cases was affected by the number of resident populations depending on the geographical location of the regions. In addition, tourist flow was another crucial factor affecting the spread of Covid-19 cases too. The vast majority of the Covid-19 cases were in the north of Italy which is a result of high population, intense tourist flow and proximity to borders of other European countries. The parameters of total Covid-19 cases, number of tourists and total population are found to have strong correlation for the top five regions with the most Covid-19 cases. Visual data analysis is a powerful technique to explore the factors that caused the spread of Covid-19 cases occurring in Italy. Additionally, forecasting models may provide valuable and essential support. Hence advanced systems may help sustainable healthcare services to be organized and executed efficiently in fighting against pandemic situations globally. Statistical analysis used in the study has shown that forecasting methods are successful in predicting Covid-19 spread which may motivate further research to extend the scope of the current findings. In addition, different models and a wider data set would be utilized to analyze the tourism factor in future studies.
Open Research
DATA AVAILABILITY STATEMENTThe data that support the findings of this article are openly available in Italy Covid-19 data set at https://github.com/pcm-dpc/Covid-19, reference numbers [39, 40], and in Istat Statistics Data Warehouse at http://dati.istat.it/Index.aspx?lang=en, reference number [41].
Supporting Information
| Filename | Description |
| cpe6774-sup-0001-supinfo.docxapplication/unknown, 591.4 KB | Data S1 Supplementary Material |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
1. WHO. Coronavirus disease (COVID-19) outbreak - about the virus; 2020. [Online] Accessed June 18, 2020. http://www.euro.who.int/en/health-topics/health-emergencies/coronavirus-covid-19/novel-coronavirus-2019-ncov
2. WHO. WHO statement on novel coronavirus in Thailand; 2020. [Online] Accessed June 18, 2020. https://www.who.int/news-room/detail/13-01-2020-who-statement-on-novel-coronavirus-in-thailand.
3. WHO. Novel coronavirus(2019-nCoV) situation report – 10; 2020. [online] Accessed June 18, 2020. https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200130-sitrep-10-ncov.pdf
4. WHO. Novel coronavirus(2019-nCoV) situation report – 11; 2020. [Online] Accessed June 18, 2020. https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200131-sitrep-11-ncov.pdf
5. Giovanetti M, Benvenuto D, Angeletti S, Ciccozzi M. The first two cases of 2019-nCoV in Italy: where they come from? J Med Virol. 2020; 92(5): 518- 521.
6. Australian T. Maybe we can fashion a viral narrative by following the workers from Wuhan to Milan; 2021. [Online] Accessed October 15, 2021. https://www.theaustralian.com.au/commentary/letters/maybe-we-can-fashion-a-viral-narrative-by-following-the-workers-from-wuhan-to-milan/news-story/32e1ff0ec5dd486e7f1b62736f8f4684
7. Casini L, Roccetti M. Reopening Italy's schools in September 2020: a Bayesian estimation of the change in the growth rate of new SARS-CoV-2 cases. medRxiv; 2021.
8. Post TW. Italy extends coronavirus lockdown to entire country, imposing restrictions on 60 million people; 2021. [Online] Accessed October1, 2021.
9. D'emilio F. Big jump in Italy's daily new cases driven by travel; 2021. [Online] Accessed October1, 2021.https://medicalxpress.com/news/2020-08-big-italy-daily-cases-driven.html
10. WHO. Weekly epidemiological update; 2021. [Online] February 23, 2021. https://www.who.int/publications/m/item/weekly-epidemiological-update---23-february-2021
11. Sharifi A, Khavarian-Garmsir AR. The COVID-19 pandemic: impacts on cities and major lessons for urban planning, design, and management. Sci Total Environ. 2020; 142391.
12. Rader B, Scarpino SV, Nande A, et al. Crowding and the shape of COVID-19 epidemics. Nature Med. 2020; 26(12): 1829- 1834.
13. Hamidi S, Sabouri S, Ewing R. Does density aggravate the COVID-19 pandemic? early findings and lessons for planners. J Am Plan Assoc. 2020; 86(4): 495- 509.
14. Sun Z, Zhang H, Yang Y, Wan H, Wang Y. Impacts of geographic factors and population density on the COVID-19 spreading under the lockdown policies of China. Sci Total Environ. 2020; 746:141347.
15. Sannigrahi S, Pilla F, Basu B, Basu AS, Molter A. Examining the association between socio-demographic composition and COVID-19 fatalities in the European region using spatial regression approach. Sustain Cities Soc. 2020; 62:102418.
16. Perone G. The determinants of COVID-19 case fatality rate (CFR) in the Italian regions and provinces: an analysis of environmental, demographic, and healthcare factors. Sci Total Environ. 2021; 755:142523.
17. Di Saverio S, Pata F, Gallo G, et al. Coronavirus pandemic and colorectal surgery: practical advice based on the Italian experience. Colorectal Dis. 2020; 22(6): 625- 634.
18. Di Saverio S, Khan M, Pata F, et al. Laparoscopy at all costs? Not now during COVID-19 outbreak and not for acute care surgery and emergency colorectal surgery: a practical algorithm from a hub tertiary teaching hospital in Northern Lombardy, Italy. J Trauma Acute Care Surgery. 2020; 88(6): 715.
19. BenÍtez CY, Pedival AN, Talal I, et al. Adapting to an unprecedented scenario: surgery during the COVID-19 outbreak. Revista do Colégio Brasileiro de Cirurgiões. 2020; 47.
20. Kapoor A, Guha S, Das MK, Goswami KC, Yadav R. Digital healthcare: The only solution for better healthcare during COVID-19 pandemic? 2020.
21. Kubota Y, Shiono T, Kusumoto B, Fujinuma J. Multiple drivers of the COVID-19 spread: the roles of climate, international mobility, and region-specific conditions. PloS One. 2020; 15(9):e0239385.
22. Wells CR, Sah P, Moghadas SM, et al. Impact of international travel and border control measures on the global spread of the novel 2019 coronavirus outbreak. Proc Natl Acad Scie. 2020; 117(13): 7504- 7509.
23. Coelho MTP, Rodrigues JFM, Medina AM, et al. Exponential phase of covid19 expansion is driven by airport connections. medRxiv 2020.
24. Gössling S, Scott D, Hall CM. Pandemics, tourism and global change: a rapid assessment of COVID-19. J Sustain Tourism. 2020; 29(1): 1- 20.
25. Baum T, Hai NTT. Hospitality, tourism, human rights and the impact of COVID-19. Int J Contemp Hospital Manag. 2020.
26. Roccetti M, Casini L. The role of inter-regional tourism in the spread of COVID-19 in Italy during the 2020 summer: a confirmatory study. Proceedings of the Conference on Information Technology for Social Good; 2021:1-6.
27. Farzanegan MR, Gholipour HF, Feizi M, Nunkoo R, Andargoli AE. International tourism and outbreak of coronavirus (COVID-19): a cross-country analysis. J Travel Res. 2021; 60(3): 687- 692.
28. Kosara R. Presentation-oriented visualization techniques. IEEE Comput Graph Appl. 2016; 36(1): 80- 85.
29. Bikakis N. Big data visualization tools; 2018. arXiv preprint arXiv:1801.08336.
30. Galván-Tejada CE, Zanella-Calzada LA, Villagrana-Bañuelos KE, et al. Demographic and comorbidities data description of population in Mexico with SARS-CoV-2 infected patients (COVID19): an online tool analysis. Int J Environ Res Public Health. 2020; 17(14): 5173.
31. Tebé C, Valls J, Satorra P, Tobías A. COVID19-world: a shiny application to perform comprehensive country-specific data visualization for SARS-CoV-2 epidemic. BMC Med Res Methodol. 2020; 20(1): 1- 7.
32. Comba JL. Data visualization for the understanding of COVID-19. Comput Sci Eng. 2020; 22(6): 81- 86.
33. Kodge B. A review on current status of COVID19 cases in Maharashtra state of India using GIS: a case study. Spatial Inf Res. 2021; 29(2): 223- 229.
34. Wieczorek M, Siłka J, Woźniak M. Neural network powered COVID-19 spread forecasting model. Chaos Soliton Fract. 2020; 140:110203.
35. Pustokhin DA, Pustokhina IV, Dinh PN, Phan SV, Nguyen GN, Joshi GP. An effective deep residual network based class attention layer with bidirectional LSTM for diagnosis and classification of COVID-19. J Appl Stat. 2020; 1- 18.
36. Achterberg MA, Prasse B, Ma L, Trajanovski S, Kitsak M, Van Mieghem P. Comparing the accuracy of several network-based COVID-19 prediction algorithms. Int J Forecast. 2020.
37. Tian T, Zhang J, Lin S, et al. Data-driven analysis of the simulations of the spread of COVID-19 under different interventions of China. J Appl Stat. 2021; 1- 14.
38. Mandal M, Jana S, Nandi SK, Khatua A, Adak S, Kar T. A model based study on the dynamics of COVID-19: prediction and control. Chaos Soliton Fract. 2020; 136:109889.
39. Morettini M, Sbrollini A, Marcantoni I, Burattini L. Covid-19 in Italy: dataset of the Italian civil protection department. Data Brief. 2020; 30:105526.
40. Protezione Civile dD. COVID-19: COVID-19 Italia - Monitoraggio situazione; 2021. [Online] Accessed February 26, 2021. https://github.com/pcm-dpc/COVID-19
41. Statistics I. Istat statistics - data warehouse; 2021. [Online]. Accessed March 15, 2021. http://dati.istat.it/Index.aspx?lang=en
42. Tang X, Dai Y, Wang T, Chen Y. Short-term power load forecasting based on multi-layer bidirectional recurrent neural network. IET Gener Transm Distrib. 2019; 13(17): 3847- 3854.
43. Xm Y, Wz F, Wang H, Chu Q, Chen Q. An attention mechanism and multi-granularity-based Bi-LSTM model for Chinese Q&A system. Soft Comput. 2020; 24(8): 5831- 5845.
44. Lee CM, Ko CN. Short-term load forecasting using lifting scheme and ARIMA models. Expert Syst Appl. 2011; 38(5): 5902- 5911.
45. Yuan C, Liu S, Fang Z. Comparison of China's primary energy consumption forecasting by using ARIMA (the autoregressive integrated moving average) model and GM (1, 1) model. Energy. 2016; 100: 384- 390.
46. Akça E, Yozgatlıgil C. Mutual information model selection algorithm for time series. J Appl Stat. 2020; 47(12): 2192- 2207.
47. Ferrara FM. Premessa. Premessa. 2020; 1- 2.
48. Conti S, Ferrara P, Fornari C, et al. Estimates of the initial impact of the COVID-19 epidemic on overall mortality: evidence from Italy. ERJ Open Res. 2020; 6(2).
49. Sun S, Xie Z, Yu K, Jiang B, Zheng S, Pan X. COVID-19 and healthcare system in China: challenges and progression for a sustainable future. Glob Health. 2021; 17(1): 1- 8.
50. Kaim A, Siman-Tov M, Jaffe E, Adini B. From isolation to containment: perceived fear of infectivity and protective behavioral changes during the COVID-19 vaccination campaign. Int J Environ Res Public Health. 2021; 18(12): 6503.
51. Mirri S, Delnevo G, Roccetti M. Is a COVID-19 second wave possible in Emilia-Romagna (Italy)? forecasting a future outbreak with particulate pollution and machine learning. Computation. 2020; 8(3): 74.
52. Ceylan Z. Estimation of COVID-19 prevalence in Italy, Spain, and France. Sci Total Environ. 2020; 729:138817.
53. Punn NS, Sonbhadra SK, Agarwal S. COVID-19 epidemic analysis using machine learning and deep learning algorithms. MedRxiv; 2020.
54. Moftakhar L, Mozhgan S, Safe MS. Exponentially increasing trend of infected patients with COVID-19 in Iran: a comparison of neural network and ARIMA forecasting models. Iran J Public Health. 2020; 49(Suppl 1): 92- 100.
55. Chimmula VKR, Zhang L. Time series forecasting of COVID-19 transmission in Canada using LSTM networks. Chaos Soliton Fract. 2020; 135:109864.
56. Kırbaş İ, Sözen A, Tuncer AD, Kazancıoğlu FŞ. Comparative analysis and forecasting of COVID-19 cases in various European countries with ARIMA, NARNN and LSTM approaches. Chaos Soliton Fract. 2020; 138:110015.
57. Shahid F, Zameer A, Muneeb M. Predictions for COVID-19 with deep learning models of LSTM, GRU and Bi-LSTM. Chaos Soliton Fract. 2020; 140:110212.
58. Tian Y, Luthra I, Zhang X. Forecasting COVID-19 cases using Machine Learning models. MedRxiv; 2020.
59. Huang C, Wang Y, Li X, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet. 2020; 395(10223): 497- 506.
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer
© 2021. Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the associated terms available at https://novel-coronavirus.onlinelibrary.wiley.com/
Abstract
At the beginning of 2020, the new coronavirus disease (Covid-19), a deadly viral illness, is declared as a public health emergency situation by WHO. Consequently, it is accepted as pandemic that affected millions of people worldwide. Italy is one of the most affected countries by Covid-19 disease among the world. In this article, our main goal is to investigate the effect of intensity of Covid-19 cases based on the population size and tourism factors in certain regions of Italy by visual data analysis. The regions of Lombardia, Veneto, Campania, Emilia-Romagna, Piemonte are the top five regions covering 58.50% of the total Covid-19 cases diagnosed in Italy. It has been shown by visual data analysis that population and tourism factors play an important role in the spread of Covid-19 cases in these five regions. In addition, a prediction model was created using Bi-LSTM and ARIMA algorithms to forecast the number of Covid-19 cases occurring in these five regions in order to take early action. We can conclude that these northern regions have been affected mostly by Covid-19 and the distribution of the resident population and tourist flow factors affected the number of Covid-19 cases in Italy.
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer




