Content area
Purpose
This paper aims to propose DisDSS: a Web-based smart disaster management (DM) system for decision-making that will assist disaster professionals in determining the nature of disaster-related social media (SM) messages. The research classifies the tweets into need-based, availability-based, situational-based, general and irrelevant categories and visualizes them on a web interface, location-wise.
Design/methodology/approach
It is worth mentioning that a fusion-based deep learning (DL) model is introduced to objectively determine the nature of an SM message. The proposed model uses the convolution neural network and bidirectional long short-term memory network layers.
Findings
The developed system leads to a better performance in accuracy, precision, recall, F-score, area under receiver operating characteristic curve and area under precision-recall curve, compared to other state-of-the-art methods in the literature. The contribution of this paper is three fold. First, it presents a new covid data set of SM messages with the label of nature of the message. Second, it offers a fusion-based DL model to classify SM data. Third, it presents a Web-based interface to visualize the structured information.
Originality/value
The architecture of DisDSS is analyzed based on the practical case study, i.e. COVID-19. The proposed DL-based model is embedded into a Web-based interface for decision support. To the best of the authors’ knowledge, this is India’s first SM-based DM system.
1. Introduction
The automatic identification of information in disaster management (DM) is one of the most compelling global problems, as countries worldwide have witnessed a significant increase in the intensity of disasters. With the spiraling coronavirus disease, i.e. COVID-19, it has become indispensable to extract and disseminate accurate and timely information. The World Health Organisation declared the coronavirus disease outbreak as a pandemic on March 11, 2020.
The situation worsened day by day, and therefore, speedy and on-time information retrieval is imperative. Social distancing, lockdowns, travel bans, self-quarantines and business closures have forced people to glue to social media (SM) more than ever before (Zhang et al., 2020). SM’s real-time data production capability makes data enormous and diverse. However, only a tiny fraction of the content is meaningful and relevant. SM data can serve as a valuable channel for seeking help, offering assistance, situational awareness, general opinions and coordinating activities in disaster. At the system level, understanding the needs, availabilities, situational updates and public views enhances the planning and rescue operations, improving disaster resilience.
Timely determination in DM is vital, but it still is challenging. Deep learning (DL) algorithms are an important part of DM systems using SM data (Caragea et al., 2016; Huang et al., 2020a; Nguyen et al., 2019). Regarding the application of SM data for DM, research suggests that the lack of tools for managing SM data during a disaster makes it difficult for disaster professionals, not making them to understand how SM data can be helpful for the public. To fill this gap in the literature, this paper explores disaster data using DL techniques to determine the nature of a SM message. Our key original contributions are as follows:
A new SM-based COVID-19 data set with the label of nature of message was developed from April 22, 2021, to May 05, 2021, with 1,03,839 tweets in total.
A new fusion model was proposed for determining the nature of a disaster-related SM message by integrating the structure of CNN and bidirectional long short-term memory network (BiLSTM).
The proposed fusion model is benchmarked against other state-of-the-art models and experimental results of previous research studies.
We demonstrate the technical efficacy by determining the nature of SM messages on the COVID-19 disaster data set.
We propose a Web-based interface to visualize the structured information, where policymakers and decision-makers can use this information to efficiently manage catastrophic events.
2. Related works
There are recently published papers on the usage of SM for DM using DL techniques. Nguyen et al. (2019) forecast people’s needs considering Hurricane disasters, using LSTM and CNN models. The models are implemented on Hurricane Sandy’12, Hurricane Harvey’17 and Hurricane Irma’17 tweets. Ofli et al. (2020) use CNN architecture on a real-world disaster-related data set gathered from Twitter, i.e. CrisisMMD for disaster response. Muhammad et al. (2019) propose a computationally efficient CNN-based architecture for fire detection, localization and semantic understanding of fire scenes.
Caragea et al. (2016) identify the flood informative messages using CNN from SM data. Pereira et al. (2020) assess the severity of floods by exploring SM data with deep CNN. Tian et al. (2018) use the LSTM model to obtain sequential semantics for disaster-related classification. Zhai et al. (2020) adopt sentiment analysis to assess Hurricane Florence’s situational awareness, using the CNN and latent Dirichlet allocation (LDA) approach. Huang et al. (2020b) explore disaster-related tweet posts for rapid disaster response, using CNN architecture.
In Table 1, we explain the summary for studies relevant to the COVID-19 disaster, with SM-based papers using DL models on COVID-19-related data sets. We have tabulated the reference of the paper, published year, main discussed content of the paper, results and limitations of the paper. Naseem et al. (2021) analyze the COVID-19 tweets, along with presenting a new large-scale sentiment data set CovidSenti, consisting of 90,000 tweets from February 2020 to March 2020. The authors classify the sentiment of COVID-19 tweets using CNN, BiLSTM, support vector machine (SVM), decision tree (DT), Naive Bayes (NB) and random forest (RF), achieving an accuracy of 0.869. Behl et al. (2021) use the Twitter platform for disaster response through sentiment analysis, considering COVID-19 and the natural hazard crises. The study achieves an accuracy of 83% with multilayer perceptron (MLP), CNN and logistic regression (LR).
Chakraborty et al. (2020) analyze the sentiments of COVID-19 tweets considering multinomial NB, SVM, LR, RF and Adaboost. The proposed model achieves an accuracy of 81%. Jelodar et al. (2020) classify the sentiments and discovers topic in coronavirus online discussions on Reddit. It achieves an accuracy of 81.5% with LSTM and LDA. Abdelminaam et al. (2021) detect fake news on Twitter automatically, using modified-LSTM and modified-gated recurrent units (GRU). The authors achieve an accuracy of 83.32%, Precision (P) of 84.45%, Recall (R) of 83.32% and F1-score of 83.7%.
Sunitha et al. (2022) analyze the sentiments of SM users during the COVID-19 disaster. It considers the Twitter platform and SM users of India and European states. It uses the LDA technique and achieves an accuracy of 95.2%, P value of 98.32%, R of 97.79%, F-score of 96.65% and Mathew’s correlation coefficient of 97.7%. Zulfiker et al. (2022) conducted sentiment analysis on COVID-19 vaccination on SM. It uses the LDA technique on tweets and achieves an accuracy of 87.44%, P = 89.62% and an F-score of 88.89%.
There is a dearth of automated SM-based DM systems. However, after various cataclysms, a few applications have been developed. In Table 2, we explain the existing SM-based systems of DM. We have tabulated the reference of the paper, published year, the main aim of the paper, platform considered, methods used, results, limitations, data sets and country of implementation of the system. Yasin Kabir et al. (2020) present STIMULATE, i.e. a real-time information acquisition and learning platform for DM. It uses the Twitter platform, considering Hurricane Harvey and Irma data sets, and is deployed in the USA. It uses DL techniques to develop the classifier. Morrow et al. (2011) explore the Ushahidi platform, which uses the Twitter platform after the earthquake Haiti in 2010. Emergency situations can be tracked using the platform with a geographical map.
Irmamiami [1] is an application to depict different activities in the Miami area related to Hurricane Irma. The platform considers Twitter, Web, email and SMS to generate disaster-based reports. It uses the Ushahidi source code. Wladdimiro et al. (2016) propose a platform to support real-time data in disaster scenarios with the Twitter platform. The authors build a case study to detect the need-based messages in an earthquake in Santiago de Chile. Imran et al. (2014) present a platform to classify the tweets as informative or non-informative, considering the 2013 Pakistan earthquake and deployed in Qatar. Burel and Alani (2018) introduce an open-source Web API, i.e. Crisis Event Extraction Service, for the automatic classification of crisis-related events from SM.
3. Research gaps and novelty of proposed work
There are a few observed issues and potential pitfalls in interpreting recently published work. Considering the COVID-19 catastrophe, the existing works focus on sentiment analysis or fake news detection. However, there is a lack of attention to determine the nature of the SM message, due to which the most significant messages, i.e. help-seeking and help-offering, are not explored. The literature lacks a Web interface for the investigation of SM-based messages for using the nature of the message. In addition, the existing literature works have specified their analysis as specialized to particular regions and cannot be generalized to those approaches globally. Although these applications provide partial functionalities to collect and process the SM data, they lack decisive components for an automatic SM-based DM system that can detect help-seeking people, assistance offering people from SM posts and update regarding situational information as well as general opinions location-wise. Hence, it is difficult to gain an understanding of the disaster situation according to this perspective. Therefore, the motivation of this paper is to propose DisDSS, which addresses this issue with a Web application. Consequently, a novel Web-based interface is developed to visualize the practical results.
4. Methodology
In this section, we propose DisDSS: a smart Web-based DM system that can be explored to determine the nature of an SM message from the Twitter data set. DisDSS is a Web-based DM system, a visual application designed to support need as well as availability-based information, situational information and general views regarding the COVID-19 disaster. The general architecture of DisDSS is illustrated in Figure 1. The following subsections explain the methodology in detail.
Sample and data
The proposed DisDSS considers the Twitter platform to extract data for SM message determination. The data is fetched using Twitter search application programming interface via Python language with the keyword #Covid. The data contains numerous features such as created at, text, lang, id and user object. Further, the user object collected features are screen name, userid, id str, name, location, URL, description, verified, followers count, friends count, listed count, favorites count, statuses count, created at and profile image URL. We gathered tweets from April 22, 2021, to May 05, 2021, with 1,43,340 tweets in total.
Manual annotation
In order to annotate the data, we de-duplicate the data set. Only unique messages are considered for annotation, i.e. 1,03,839 messages. We label the data set into five categories:
need-based;
availability-based;
situational-based;
general; and
irrelevant.
We show the annotation scheme with the class label below:
-
1.Need-Based SM messages: messages those contain information about the need or unattainable resources, such as ICU beds, oxygen cylinders, vaccination, food, water, electricity, etc.
-
2.Availability-based SM messages: messages that inform about the availability of certain resources. This class comprehends the potentially available as well as actually available resources.
-
3.Situational SM messages: tweets that provide information related to the number of cases, deaths, injuries, etc.
-
4.General SM messages: the class embodies tweets that are mentioning about prayers, cautions, advice, emotional support, criticizing, etc.
-
5.Irrelevant SM messages: tweets that contain no information related to the disaster.
Data pre-processing
The collected data set is raw, noisy, unstructured and informal. The next step in the development of DisDSS is to pre-process the data. To do so, the raw data is preprocessed by removing retweets and eliminating dispensable features. All the features are banished except tweet no, text and manual label. The hyperlinks and URLs are removed. Non-English characters, along with additional white spaces, are removed. The tweets are folded into lowercase and tokenized into smaller units called tokens. Further, English language stop words are removed.
Modeling
We randomly split the data set into three subsets:
training;
validation; and
testing in the ratio of 60:20:20.
which is a commonly used split ratio (Alam et al., 2018; Tam et al., 2021). The training data is used to train the model, and the validation data set is used to tune the hyperparameters and provide an unbiased evaluation to select the best model. The testing data set is used for testing and prediction purposes. Figure 2 depicts the distribution of covid data set and the workflow of training, validation and testing data sets. It is worth noting that we save the trained model from the validation stage and call the saved model for the testing stage.
In this paper, we develop a DL-based hybrid model, which is composed of multiple layers of the cascade. Figure 3 elucidates the proposed model, encapsulating the layers. The model consists of a CNN layer, which receives the word embeddings for each token in the tweet as inputs. Subsequently, the convolution layer’s output is pooled to a smaller dimension. The output of the pooling layer is fed into the BiLSTM layer, where it extracts the context information of words. Dropout layers are used to overcome the over-fitting issues. The dense layer ultimately outputs the tweet as need-based, availability-based, situational-based, general-based or irrelevant messages using a softmax function. The pseudo-code for the proposed model is summarized in Algorithm 1.
Integrating CNN, and BiLSTM layers, not only reaps the benefits of CNN extracting local features but also takes into account the advantages of BiLSTM in contextual information of text sequences. BiLSTM is used as it uses both forward LSTM and backward LSTM. LSTM considers only past information, ignoring future information. Moreover, not all the parts of the SM message are equally relevant, but LSTM cannot recognize the different relevance between each part of the message. Our proposed model is relatively complex, using CNN and BiLSTM layers. From a linguistic point of view, BiLSTM considers the meaning of the words in context and overcomes the drawback that LSTM does not consider the information after words (Liu et al., 2020). The parameter settings for the proposed model are shown in Table 3.
The working environment of the proposed approach is as follows:
Hardware configuration: Intel(R) Core(TM) i5-8250 8th Generation, 8GB RAM, 512 GB SSD and NVIDIA GeForce MX150.
Software configuration: Microsoft Windows 10, 64-Bit, Python 3.8.3.
Python is a high-level, general purpose programming language that is used to interact with DL libraries as application program interfaces. The experiments are carried out using open-source libraries such as NumPy, Pandas, TensorFlow and Keras (DL framework):
Web interface
Front-end: creating the layout of the Web platform.
We recognize that scientists and disaster professionals with knowledge of DM but limited computer programming skills are potential users of this interface. For these users, we use HTML, which is compatible with most recent Web browsers on computers, laptops, etc. Hyper-text markup language (HTML) is a standard markup language used to describe the structure and provide a design for the various documents to be displayed in the Web browser. With the recent merging of technologies such as cascading style sheet, the visualization of Web browsers has augmented substantially. The embedding of HTML, images, various forms, etc., is done into a Web browser considering the user’s needs. The most updated HTML 5 is used to produce a lightweight user-friendly interface. The URL for the browser is http://127.0.0.1:5000.
Back-end: setting up data storage and enabling graphical user interface.
The back end takes care of the storage of the COVID-19 database and, in addition, controls the graphical user interface. This is brought about by comma-separated values and Python. All core model functions are written in Python language. Python offers several advantages over other languages, as it is open-source, free and lightweight. Flask framework, a lightweight python framework, is used for Web development.
5. Data analysis
To evaluate the performance of our model, the focus is not only on one performance metric; rather, multiple metrics are considered. Accuracy, P, R and F1-score are the performance metrics. Furthermore, the Area under ROC and Area under PR-curve are deployed to measure the completeness and robustness of the framework. The performance metrics used in this research study are defined in Table 4.
Even though there is no universally accepted cutoff point for aforementioned performance measures such as accuracy, precision, recall and F-score, it is widely acknowledged that the higher the value of metrics, the higher the performance. The best and worst values of accuracy, P, R, F-score, AU-ROC and AU-PR-curve are 1, 0 (Luo and Xu, 2021).
6. Results
In this section, we present the word cloud for each human-annotated class in Figure 4. The word clouds a, b, c, d and e correspond to the class label 0, 1, 2, 3 and 4, respectively. The class distribution in Figure 5 represent the need-based, availability-based, situational-based, general and irrelevant messages corresponding to class 0, 1, 2, 3 and 4.
We build and train DL models – CNN, LSTM and BiLSTM. The comprehensive descriptions of CNN, LSTM and BiLSTM are narrated in Kim (2014), Hochreiter and Schmidhuber (1997) and Goodfellow et al. (2016), respectively. Their general architectures, generated through the Anaconda-integrated platform are displayed in Figure 6(a), 6(b) and 6(c), correspondingly. The architecture of the proposed model is showcased in 6d. To further evaluate the performance, we compare our proposed approach with CNN, LSTM and Bi-LSTM. Table 5 shows the performance results of the conducted experiments in tabular format, and Figure 7 depicts the results pictorially.
Accuracy vs epochs plots are illustrated in Figure 8. The x-axis represents the number of epochs, and the y-axis represents the learning curve with the scaling (0,1). We can observe that the accuracy of the proposed model is 0.91, i.e. 91% in Figure 8(d), which is higher than 0.87 of CNN [Figure 8(a)], 0.87 of LSTM [Figure 8(b)] and 0.90 of BiLSTM [Figure 8(c)]. Accuracy is the percentage of correct classifications made by the classifier. The sum of true positive (TP) and true negatives (TN), divided by the grand total of TP, TN, false positive (FP) and false negative (FN), is calculated as accuracy. In the proposed model, it reaches an accuracy higher than 90% in both training and validation sets. Both the training accuracy curve and validation accuracy curve are depicted in Figure 8(d) levels. It is shown that the curves stabilize; hence, indicating no significant overfitting problem.
As can be seen from Column III, Table 5, the P of the proposed model is 0.90, CNN has 0.87, LSTM has 0.88 and BiLSTM has 0.90. The proportion of all TP votes and the grand total TP with FP votes is “precision.” A higher value of P of the pro-posed approach signifies the reliability of the model and acts as a measure of the exactness of the results. A higher value of R of the proposed approach (0.90) as compared to other models (0.86 of CNN, 0.88 of LSTM and 0.89 of BiLSTM) is showcased in Column IV, Table 5. The proportion of the TP votes and the grand total of TP with FN votes is “recall.” The R metric is a measure of completeness. It means that the proposed approach returns the relevant results. In Column V, Table 5, the F-score is displayed. The F-score value of the proposed approach (0.90) is higher than all the models (0.86 for CNN, 0.88 for LSTM and 0.89 for BiLSTM). F-score is the harmonic mean of P and R. It takes into account both P and R, i.e. both FP and FN. In addition, harmonic means of the F-score penalize unequal values more and punish extreme values (Luo and Xu, 2021).
We calculate ROC and micro-averaged the performance. ROC curve using CNN, LSTM and BiLSTM architecture is shown in Figures 9(a), (b) and (c), respectively, taking into account TP rate and false positive (FP) rate. The ROC curve is attractive because of the insensitivity toward changes in class distribution. AU-ROC is a measure of the goodness-of-fit of the models on the training data set and the prediction capability of the models on the validation data set. The closer the value is to 1, the better the model (Tien Bui et al., 2016). The ROC curve for our proposed model, implemented on the covid data set, is illustrated in Figure 9(d). Our proposed model performs best amongst all the models on the area under ROC metric of the proposed model is 0.97, which is larger than that of the others (0.96, 0.96, 0.96).
Figure 10 plots the PR-curve of all DL models. The area under PR-curve is a measure based on PR-curve, i.e. a plot of P(y-axis) and the R(x-axis). The higher the area under PR-curve, the better the model (Tang et al., 2019). The AU-PR-curve of the proposed model is 0.91 [Figure 10(d)], which is higher than CNN [Figure 10(a)], LSTM [Figure 10(b)] and BiLSTM [Figure 10(c)]. We can interpret that the AU-PR-curve of the proposed model is 2.24% higher than all the other models. Hence, considering all the evaluation metrics, we can observe that the proposed approach is better than CNN, LSTM and BiLSTM.
7. DisDSS system architecture
This section introduces DisDSS: the Web interface for the visualization of experimental results. DisDSS obtains the resultant data from the proposed model, stored in CSV format.
The main aim of the DisDSS is to make the system easily accessible and comprehensive for use by disaster professionals. The DisDSS is integrated by merging two modules:
front-end: to build the basic layout of the system; and
back-end: to store the disaster-related database and enable a graphical user interface.
Detailed descriptions of these two modules are provided in Section 4.5. More specifically, the tool consists of four main pages:
Home Page: The first page, which runs, is the homepage. The page is to select the concerned country’s data. The user selects the country using the drop-down list. Figure 11 illustrates the graphical user interface for the selection of country.
State and Metropolitan City Selection: States of India are drop-down upon India’s selection as a country. Figure 12 showcases the graphical user interface for state wise/metropolitan selection.
Type of Data Selection: The user can select the type of data they want to see, i.e. need-based, availability-based, situational-based, general or irrelevant. The user can directly click on the respective button to visualize the results. Figure 13 depicts the graphical user interface showing need-based, availability-based, situational-based, general and irrelevant messages.
Data Visualisation: DisDSS showcases the results in the tabular format. The table provides the tweet no., tweet text, name of the tweet user, and location.
Figure 14 illustrates the graphical user interface visualization results.
8. User experience
In this section, we provide usage scenarios that demonstrate DisDSS’s effectiveness and usability.
Usage scenario 1
The user is interested in identifying people in need of help during COVID-19 in the Delhi area. Therefore, the user selects India in the country drop-down, subsequently selecting Delhi in state/metropolitan drop-down. After applying the Delhi filter, the user clicks the need-based button to see the need-based SM messages. Figure 15 illustrates the output of the Delhi region’s need-based SM messages.
The user explores the filtered messages in Figure 15 and finds messages that say (Tweet no. 96,500, Row 1) urgently in need of an icu bed in Delhi, Ghaziabad, Noida, Gurgaon. Tweet no. 94,548, Row 2 states that Name: Rajnish Jain Area: Rani Bagh Oxygen Level 90 Age: 38 Urgently Bed Required in Delhi. Similarly, Tweet no. 97,517, penultimate row in Figure 15, mentions about the need of oxygen cylinders in Sar Gangaram Hospital Delhi, with more than 500 patients admitted. After further finding the tweet messages in Figure 15, the user finds misclassifications as well. Tweet no. 95,885, Row 4 states CharuPragya Anyone can please help with a source for a bed in medanta #Lucknow?? This is urgent. The SM message is asking for help in Lucknow and not Delhi.
Usage scenario 2.
To demonstrate the diverse functionality of the DisDSS, we apply the filter at the availability-based button for India. Figure 16 showcases the availability-based SM messages of India. Tweet no. 80,869, Row 1, states the verified oxygen availability in Delhi with mobile number. Furthermore, Tweet no. 92,172 (Row 2), 84,313 (Row 3), 84,241 (Row 6) and 72,954 (Row 7) mention the details about icu beds availability at Indore and Lucknow. The misclassifications are depicted in a tweet no. 9,163 (Row 4) and 11,138 (Row 8).
User scenarios demonstrate the overcoming of existing systems in DisDSS. The easy-to-use Web interface helps in exploring the significant messages, using the nature of SM messages. DisDSS gets better off by expanding the approach at the global level, considering different countries/states/union territories. It contains components to achieve an understanding of the disaster situation and make effective decisions.
9. Discussion
In this research, we develop a Web-based DM system to determine the nature of the SM message, considering COVID-19 disaster. The research proposes a hybrid deep learning model to classify the tweets into need-based, availability-based, situational-based, general and irrelevant categories. The proposed hybrid model performs better than the state-of-the-art CNN, LSTM and BiLSTM, considering six performance metrics, i.e. accuracy, P, R, F-score, ROC and PR-curve.
The Web-based platform is developed using front-end and back-end layouts. The developed DM system transforms the raw and unstructured data into disaster-relevant information in a structured format. The findings of the research aid disaster professionals in efficient DM.
The existing literature focuses on sentiment analysis and fake news detection on the COVID-19 disaster. The current research works on identifying the nature of the SM message. It proposes a new COVID-19 data set with the nature of the message label. The existing research is limited to a specific region implementation. DisDSS further supports location wise classification. It provides the country-wise, state-wise and metropolitan cities data as per requirement. It is useful to gain an understanding of the disaster scenario. The extension of this work, DisDSS 2.0, has been presented at European Geoscience Union General Assembly 2022 (Singla et al., 2022).
10. Implications
The information can be obtained from Twitter data gathered using Twitter Search Application Programming Interface using DL techniques. The proposed DL technique is a hybrid model of CNN and BiLSTM layers. The proposed model uses the synergies of both individual models.
Existing COVID-19 literature studies focus on sentiment analysis or fake news detection. The current study determines the nature of SM messages in the context of DM. In addition, the introduction of a Web interface for using the nature of SM messages can be used by disaster managers to extract need-based, availability-based, situational-based, general and irrelevant messages.
Efficient DM requires automated DM systems to extract the right information so that it can be accessed by the right person and can be used at the right time. DisDSS is the system catering to the need of the hour. It helps in automatically fetching the information location-wise. The findings of the case study, COVID-19, can be used by decision-makers for effective operations of DM. It has important implications for disaster response and future efforts to improve DM.
11. Conclusion
The paper seeks to develop DisDSS, a novel platform composed of a Web browser using DL to help determine the nature of SM messages and to help disaster professionals quickly respond to disasters. Our system attempts to reduce the gap between decision-makers and SM usage for DM. The idea is to expedite their use and reinforce the decision-making process with accurate and reliable knowledge to put together correct actions to mitigate the effects of the disaster.
We develop a new covid data set with a manual label of need-based, availability-based, situational-based, general and irrelevant SM messages. In the current study, we propose a hybrid model considering the contextual information of the SM message. It is composed of CNN, and BiLSTM layers. In the empirical section, we conduct extensive experiments against different neural network models to validate the effectiveness of the proposed model. The experiments illustrate the better performance of the proposed DL model in terms of accuracy, P, R, F-score, ROC and PR-curve. To demonstrate the technical potency of the proposed model, we developed the Web interface to visualize the experimental results of the SM messages. It can be observed from the results that the model is significantly contributing to effective decision-making in the catastrophic times.
The results illustrate the user scenarios of need-based SM messages in the Delhi region. The graphical user interface of DisDSS displays the messages related to the requirements of ICU beds, oxygen cylinders and sources for a bed in a hospital. In another instance of a user scenario, the findings illustrate the availability-based SM messages of the country India, on the whole. It showcases the tweets with the potential availability of oxygen cylinders and ICU beds in the Delhi and Indore regions, respectively.
Our DisDSS is developed with user convenience in mind, influencing all of our design, computational evaluation and implementation choices. Our user-centered model and Web interface contribute to both DL and DM communities. We bridge the two fields by demonstrating how models can be trained and evaluated and used to facilitate need-based, availability-based messages as well as situational awareness for real-life, practical use.
12. Limitations and future work
Though our research contributes to knowledge, we acknowledge that it has some limitations. The study is limited, as it represents only one SM platform, Twitter, on a singular disaster topic and thus cannot be generalized to other disasters and SM platforms. It is recommended that further research could be conducted on a larger scale by incorporating more SM platforms that deal with disasters of varying natures. Therefore, comparative studies that investigate the use of other SM platforms for efficient DM are recommended.
Note1.https://irmamiami.ushahidi.io/views/map
Figure 1.Structure of DisDSS: a smart disaster management system for the nature of social media message determination
Figure 2.The trained model is saved and called upon for the testing process, which is further used for the determination of the SM message as a need-based message, availability-based message, situational-based message, general message or irrelevant
Figure 3.Proposed model encapsulating the layers
Figure 4.Word clouds of human-annotated classes
Figure 5.Covid data set message distribution after manual annotation
Figure 6.Architectures of four DL models
Figure 7.Comparison of performance metrics
Figure 8.Accuracy vs epochs plots
Figure 9.ROC plots of DL models
Figure 10.PR-curve of all DL models
Figure 11.Graphical user interface for country selection
Figure 12.Graphical user interface for state/metropolitan selection
Figure 13.Graphical user interface showcasing need-based, availability-based, situational-based, general or irrelevant messages
Figure 14.Graphic user interface for result visualization
Figure 15.Results displaying need-based messages in Delhi
Figure 16.Results displaying availability-based messages in Delhi
Table 1.
Studies pertaining to COVID-19 and SM
| Study | Aim | Platform | Methods | Results | Limitations |
|---|---|---|---|---|---|
| Naseem et al. (2021) | Sentiment analysis of COVID-19 related tweets | CNN; BiLSTM; SVM; DT; NB; and RF | Accuracy = 0.869 (86.9%) | Only accuracy metric is used for evaluation. | |
| Behl et al. (2021) | Twitter for disaster relief through sentiment analysis | MLP; CNN; and LR | Accuracy 83% | Underperformance because of the limitations on smaller training data set. | |
| Chakraborty et al. (2020) | Sentiment analysis of COVID-19 tweets | Multinomial NB; SVM; LR; RF; and Adaboost | Accuracy 81% | Study is limited to only one DL technique. | |
| Jelodar et al. (2020) | Sentiment classification and topic discovery of Coronavirus online discussions | LSTM; LDA | Accuracy 81.5% | Study limited to only one DL technique. | |
| Abdelminaam et al. (2021) | Automated detection of misleading COVID-19-related information | Modified-LSTM; and Modified GRU | Accuracy = 83.82%; |
Only one SM platform is considered. | |
| Sunitha et al. (2022) | Sentiment analysis of SM users | LDA | Accuracy = 95.2% |
Only one SM platform is considered | |
| Zulfiker et al. (2022) | Sentiment analysis on COVID-19 vaccination in social media | LDA | Accuracy = 87.44% |
Only one country is considered |
Table 2.
Existing social media-based systems for disaster management
| Study | Aim | Platform | Methods | Results | Limitations | Data sets | Country |
|---|---|---|---|---|---|---|---|
| Yasin Kabir et al. (2020) | STIMULATE – a System for Real-time Information Acquisition and Learning for DM | BiLSTM, and CNN | Classifies tweets into FEMA-defined rescue priorities | The system considers flood disasters, with need-based tweet classification | Hurricane Harvey and Irma; Nepal Earthquake; California Earthquake; Typhoon Hagupit; Cyclone PAM; CrisisLex | USA | |
| Morrow et al. (2011) | Ushahidi | – | Emergency situations can be tracked with geographical map | Limitation in information use from UHP website | Earthquake | Haiti | |
| Wladdimiro et al. (2016) | Platform to support real-time data in disaster scenario | – | Basic supplies such as water and electricity; disaster information; crimes, and looting; and missing people information | Only one disaster is considered | Earthquake | Santiago de Chile | |
| Irmamiamia | Depicts different activities in Miami area related to Hurricane Irma | Twitter; Web, email, SMS | – | Disaster-based reports | Only one disaster Hurricane is considered | Hurricane Irma | USA |
| Imran et al. (2014) | Platform to automatically classify crisis-related microblog information | – | Classification of messages as informative, and non-informative | Earthquake is the only considered disaster | 2013 Pakistan Earthquake | Qatar | |
| Burel and Alani (2018) | Crisis event extraction service (CREES) – Automatic detection and classification of crisis-related content on SM | CNN | Three types of reports: crisis vs non-crisis, type of crisis and type of information | Performance drops identifying fine-grained event-related information | CrisisLex T26 | UK |
Table 3.
The key parameters (hyperparameters) for proposed DL model
| Hyperparamter | Definition | Value |
|---|---|---|
| Epochs | Iteration count | 25 |
| Embedding Dimension | Size of vector used to represent each of the word embedding | 300 |
| Maximum Sequence length | Maximum tweet length | 23 |
| Activation Function | Calculates the weighted sum of its input, adds a bias and then decides whether to activate the neuron or not | ReLU |
| Dropout Rate | Some hidden layer neurons are discarded with the 40% probability during the training process to reduce the dependence on some local features in each iteration | 0.4 |
| Optimizer | Method used to update the weights, in order to reduce the error | Adam(1e-3) |
| Batch Size | Data is grouped into batches prior to feeding it into DL model | 1,024 |
| Loss Function | Function to assess model prediction | Categorical |
Table 4.
Measures of performance for the proposed study
| Name | Description |
|---|---|
| Accuracy = (True Positives (TP) + True Negatives (TN))/ |
% age of correct predictions by the classifier |
| P = TP/(TP + FP) | Measure of exactness(%age of predicted relevant tweets that are actually relevant) |
| R = TP/(TP+FN) = True Positive Rate (TPR) | Measure of completeness(%age of relevant tweets labeled as such) |
| F-measure = (a2 + 1)P × R/(a2 (P + R) | Highly informative measure, considering harmonic mean of P and R |
| AU-ROC | Area under ROC is a metric to measure accuracy through area under the ROC |
| AU-PRC | Area under PR-curve is a measure based on PR-curve, i.e. a plot of P(y-axis), and the R(x-axis) |
Table 5.
Comparison results for the proposed model against other classifiers
| Approach | Accuracy | Precision | Recall | F1-score | AUC | PR score |
|---|---|---|---|---|---|---|
| Proposed model | 0.91 | 0.90 | 0.90 | 0.90 | 0.97 | 0.91 |
| CNN | 0.87 | 0.87 | 0.86 | 0.86 | 0.96 | 0.89 |
| LSTM | 0.87 | 0.88 | 0.88 | 0.88 | 0.96 | 0.89 |
| Bi-LSTM | 0.90 | 0.90 | 0.89 | 0.89 | 0.96 | 0.89 |
© Emerald Publishing Limited.