Substitutability of urban sustainability assessment indicators: A semantic similarity analysis

Abstract

Urban sustainability assessment is an effective method for objectively presenting the current state of sustainable urban development and diagnosing sustainability-related issues. As the global community intensifies its efforts to implement the sustainable development goals (SDGs), the demand for assessing progress in urban sustainable development has increased. This has led to the emergence of numerous indicator systems with varying scales and themes published by different entities. Cities participating in these evaluations often encounter difficulties in matching indicators or the absence of certain indicators. In this context, urban decisionmakers and planners urgently need to identify substitute indicators that can express the semantic meaning of the original indicators and consider the availability of indicators for participating cities. Hence, this study explores the relationships of substitution between indicators and constructs a collection of substitute indicators to serve as a reference for sustainable urban development assessment. Specifically, building on a review of international and Chinese indicators related to urban sustainability assessment, this study employs natural semantic analysis methods based on the Word2Vec model and cosine similarity algorithm to calculate the similarity between indicators related to sustainable urban development. The results show that the Skip-gram algorithm with a word vector dimensionality of 600 has the best performance in terms of calculating the similarity between sustainable urban development assessment indicators. The findings provide valuable insights into selecting substitute indicators for future sustainable urban development assessment, particularly in China.

Headnote

ARTICLE INFO

Keywords:

Urban sustainability assessment

Sustainable development

Semantic analysis

Word2Vec

Cosine similarity

ABSTRACT

Urban sustainability assessment is an effective method for objectively presenting the current state of sustainable urban development and diagnosing sustainability-related issues. As the global community intensifies its efforts to implement the sustainable development goals (SDGs), the demand for assessing progress in urban sustainable development has increased. This has led to the emergence of numerous indicator systems with varying scales and themes published by different entities. Cities participating in these evaluations often encounter difficulties in matching indicators or the absence of certain indicators. In this context, urban decisionmakers and planners urgently need to identify substitute indicators that can express the semantic meaning of the original indicators and consider the availability of indicators for participating cities. Hence, this study explores the relationships of substitution between indicators and constructs a collection of substitute indicators to serve as a reference for sustainable urban development assessment. Specifically, building on a review of international and Chinese indicators related to urban sustainability assessment, this study employs natural semantic analysis methods based on the Word2Vec model and cosine similarity algorithm to calculate the similarity between indicators related to sustainable urban development. The results show that the Skip-gram algorithm with a word vector dimensionality of 600 has the best performance in terms of calculating the similarity between sustainable urban development assessment indicators. The findings provide valuable insights into selecting substitute indicators for future sustainable urban development assessment, particularly in China.

(ProQuest: ... denotes formulae omitted.)

1. Introduction

With the advancement of global urbanization, more than half of the world's population now lives in cities (Brunn et al., 2003). Urban areas are now the focal point of human economic, social, and cultural activities, accompanied by substantial resource demands and significant environmental pressures (Zhang et al., 2019). Issues, such as air pollution, water scarcity, traffic congestion, waste management, and housing inequality (Satterthwaite, 2011) directly affect the quality of life of urban residents and pose severe threats to the ecological balance and sustainability of resources worldwide. To address these challenges, the United Nations Sustainable Development Summit in 2015 adopted the "Transforming our World: the 2030 Agenda for Sustainable Development" (hereinafter referred to as "the 2030 Agenda"), which includes 17 sustainable development goals (SDGs), 169 targets, and 243 indicators, aimed at guiding cities and countries toward sustainable development (Lee et al., 2016). However, recent studies have shown that, based on most of the available data on indicators, the world is still far from achieving the goals set in 2015, posing a real risk of not meeting the SDGs by 2030 (Shulla and Leal-Filho, 2023). Therefore, the United Nations has called on all countries to "Take action in 2023", urging civil society, businesses, and other stakeholders to "Strongly support these goals" and to enhance "The global movement to achieve the goals", laying the foundation for coordinated efforts toward achieving sustainable development. In light of these developments, the global demand for assessments of the progress of sustainable urban development has intensified (Salem et al., 2020). This has led to the proliferation of indicator systems with varying scales and themes released by different evaluators, With the number of indicators on the rise (Angelidou et al., 2018; Li et al., 2009; Marvuglia et al., 2020). Owing to differences in the understanding of urban sustainability among various evaluators as well as variations in statistical systems and data availability, existing urban sustainable development evaluation indicator systems (i.e., the United Nations' global indicator framework and the localized indicator systems constructed by various countries and regions) are not fully applicable to all urban sustainable development assessments (Huan et al., 2022; Zhu et al., 2019). Consequently, addressing the issue of indicator mismatch or absence becomes a critical practical need for urban decision-makers and planners, urging them to seek substitute indicators that can express the original indicators' semantic meaning and consider the availability of indicators for the participating cities.

Traditional methods for seeking substitute indicators have mainly relied on expert assessments based on professional knowledge and experience as well as subjective similarity scoring (Nourry, 2008). These methods leverage the expertise and experience of domain experts, offering real-time, intuitive, and flexible solutions. However, they inevitably suffer from limitations related to subjectivity, scalability, and high costs (Hanley et al., 1999). With the rapid development of computer technology, natural language processing (NLP) has been widely used for the selection of substitute indicators (Chowdhary, 2020). NLP, a subset of artificial intelligence (АГ), is designed to enable computers to understand, interpret, and process natural languages. NLP technologies allow computers to interact with humans in natural languages, enabling them to extract useful information from vast amounts of textual data (Nadkarni et al., 2011). Semantic similarity analysis, an NLP task, aims to measure the degree of semantic similarity between two text segments (Harispe et al., 2015). Compared with traditional methods, approaches based on text similarity calculations can automate and efficiently process a large volume of indicator data. This not only saves time and costs but also reduces the subjectivity and human bias inherent in traditional methods, making the process more objective. Therefore, this study aims to fully leverage the advantages of NLP to explore the substitution relationships between urban sustainability assessment indicators, providing a reference for sustainable urban development assessment work. Specifically, it examines the characteristics of Chinese texts and short texts of sustainable development indicators using the Word2Vec model and cosine similarity algorithm.

2. Materials and methods

2.1. Data sources

The existing indicator systems used to assess the progress of sustainable urban development are diverse. This study collected and reviewed 1 238 indicators related to urban sustainability assessments from both international and Chinese sources. On the international scale, this study primarily focused on the SDG global indicator framework and reports related to the localization practices of the global SDGs (Adams and Judd, 2011), paying particular attention to the localized SDG11 indicator systems suitable for the urban contexts of European cities, American cities, and Deging County in Zhejiang Province, China (Abraham, 2021; Lynch et al., 2019). Within the Chinese context, the current study reviewed sustainable urban developmentrelated indicator systems released by authoritative institutions, such as Chinese government agencies and university research institutes, including the National Healthy City Evaluation System, Resilient City Index, and SDG11 Progress Assessment Indicator System (Figure 1) (Gao et al., 2019; Zhang et al., 2022).

2.2. Experimental design

The experiment was divided into two phases: determining the Word2Vec parameters and constructing the model.

Phase 1 involved calculating the similarity of the indicators within the urban sustainable development indicator database using Word2Vec and cosine similarity. By varying the parameters of the Word2Vec model, different similarity calculation results were obtained. The results were then evaluated using metrics such as precision, recall, F1 score, and accuracy to assess the model's classification capability and effectiveness. This evaluation facilitated the improvement of the model and the determination of its parameters for Phase 2.

Phase 2 involved constructing a framework for the urban sustainable development assessment indicator similarity calculation model, as shown in Figure 2. The model was divided into five parts: data loading and preprocessing, model training, text similarity calculations, interaction program, and termination program. The bestperforming Word2Vec parameters from Phase 1 were incorporated into model training to enhance the accuracy of the similarity calculation results. The model primarily relied on user-friendly interactions based on the similarity calculation results and visualized these results, providing references for urban sustainable development assessment.

2.3. Evaluation metrics

The evaluation metrics adopted were accuracy (A), precision (P), recall (R), and F1 score (F1) (Yacouby and Axman, 2020). The evaluation metrics used were defined as follows. A denotes the accuracy of the model, representing the proportion of correctly classified samples out of the total number of samples. P denotes precision, which is the ratio of true-positive samples to all samples that have been identified as positive. R denotes recall, which is the ratio of true-positive samples to all samples that are actually positive. F1 represents the harmonic mean of precision and recall. The formulas for these metrics are as follows.

... (1)

... (2)

... (3)

... (4)

Where, TP (true positives) is the number of samples correctly identified as a substitute indicator, FP (false positives) is the number of samples that are not substitutes but have been incorrectly identified as a substitute, TN (true negatives) is the number of samples correctly identified as not a substitute, and FN (false negatives) is the number of samples that are actually substitutes but have been incorrectly identified as not a substitute.

2.4. Related algorithms and models

2.4.1. Cosine similarity algorithm

Cosine similarity is particularly important in fields such as NLP, information retrieval, and recommendation systems. It can be used to perform tasks such as text matching, text clustering, and recommendation of similar texts. It is a commonly used method for measuring text similarity. It measures the similarity between two texts by calculating the cosine of the angle between their two vectors. The value ranges from -1 to 1, with values closer to 1 indicating higher similarity between the texts, and values closer to -1 indicating lower similarity (Xia et al, 2015). It is calculated as cos (7,7')=7x 7'/|T|x |T|, Where, T and T" represent the vectors of two different texts, and cos represents the cosine calculation. Hence, cos (T, T") represents the cosine distance between the two texts.

2.4.2. Word2Vec model

Word2Vec is a word vector model developed by Google in 2013, aimed at transforming words in textual data into dense vectors so that computers can better understand and process text information (Church, 2017).

2.4.2.1. Model structure

The core idea of Word2Vec is to map words into a continuous vector space by training a neural network model such that words with similar meanings and syntax are closer in the vector space (Lilleberg et al., 2015). Word2Vec includes two training approaches: the continuous bag-of-words (CBOW) and Skip-gram models (Xiong et al., 2019). CBOW predicts the target word based on context words, whereas Skip-gram predicts context words based on a target word. Both models are characterized by low complexity and high computational efficiency and can process complex corpora with a vast vocabulary. The main difference lies in their training speeds, with CBOW being faster, while Skip-gram is better suited for smaller datasets. Figure 3 shows schematic diagrams of Skip-gram and CBOW.

Skip-gram uses vectors of dimension V as the input, where N represents the number of different words, and WO is the weight matrix from the input layer to the hidden layer, with a size of Nx V. WI is the weight matrix from the hidden layer to the output layer, with a size of VXN, where each column corresponds to a word vector of an output word. The CBOW model has a similar training process, but the contents of the input and output layers are exactly opposite. The specific Skip-gram algorithm is divided into two steps: first, calculating the probability of the context words, and second, maximizing the product of all the conditional probabilities. The formula used is as follows.

... (5)

... (6)

Where, P(c|w) represents the conditional probability of the context word c appearing given the target word w; exp is the exponential function; v(c), v(c'), and v(w) represent the vector representations of the words c, c', and w, respectively; and L is the objective function of Skip-gram, which is the product of the conditional probabilities P (c|w).

Likewise, the specific CBOW algorithm is divided into two steps: first, the output of the hidden layer h is calculated, and second, the input at each node of the output layer is calculated. The input u, of each node in the output layer is obtained by multiplying the output vector h of the hidden layer with the weight matrix WO, The formula used is as follows.

... (7)

Where, C represents the size of the context window, and > indicates the sum of the vector representations of all words in the context.

2.4.2.2. Model parameters

Based on the characteristics of the experimental data and requirements of the experimental environment, this study utilized the Word2Vec tool provided by the open-source third-party Gensim library for training word vectors. The training results are influenced by the training parameters (Rong, 2014), and the training effectiveness can be improved by adjusting parameters such as the training algorithm, dimensionality of word vectors, and size of the context window. The training is set to use the CBOW model by default, which is faster than the Skip-gram model but less effective in learning because of infrequently occurring words. Therefore, this study obtained the best parameter information for feature extraction by changing the algorithm and dimensionality of the word vectors. The model parameters used in this study are listed in Table 1. Changing sg alters the algorithm, and adjusting the size changes the dimensionality of word vectors to select the best-performing dimensionality for calculating the similarity of urban sustainable development assessment indicators.

3. Results and discussion

3.1. Parameter settings and accuracy assessment

To compare and analyze the impact of different parameter settings on similarity outcomes, based on six distinct parameter configurations, the most and least similar indicators for each metric were identified, their substitutability was assessed, and the A, P, R, and F1 for each scenario were calculated. The results are summarized in Table 2. It can be observed that the worst performances in terms of A, P, and F1 score were all under the condition of size 400 and sg 1, with values of 90.41%, 80.89%, and 89.13%, respectively, and the worst R occurred at size 200 and sg 1 at 97.85%. However, when sg is set to 1 and size to 600, that is, the Skip-gram algorithm has a word vector dimensionality of 600, the model performed the best, with A of 94.01%, P of 87.98%, R of 99.65%, and F1 of 93.45%. Based on the evaluation results under different parameters, this study constructed a framework for calculating the similarity of sustainable urban development assessment indicators using the Skip-gram algorithm with a word vector dimensionality of 600.

3.2. Text preprocessing situation

Text preprocessing is a key step in text analysis, which involves the processing of text to convert raw textual data into clean, structured data suitable for further processing and analysis for subsequent NLP tasks. The main text preprocessing steps used in this study included text cleaning, stop-word removal, punctuation removal, and Chinese word segmentation. The first step involved text cleaning of the collected indicator data. The purpose of text cleaning is to remove duplicate indicators across different indicator systems to reduce redundancy and noise and improve data quality. After text cleaning the collected 1 238 indicators, 1 082 indicators remained, which were then processed to remove stop words and punctuation and for word segmentation.

First, stop words such as "de s", "in", and "and", which carry no meaningful semantic information, were removed to refine the original text data and reduce data dimensions. Subsequently, all punctuation marks that could affect subsequent text processing were also eliminated to enhance text clarity. Finally, the Jieba word segmentation tool was used to obtain the text preprocessing results (i.e., word segmentation results); Table 3 lists some examples.

3.3. Word vector training results and analysis

Based on the preprocessed indicators, word vectors were trained using the Word2Vec model, allowing the acquisition of vectorized representations of the indicator1s to be used as input data for subsequent calculations. As discussed in Section 3.1, the model had the best performance when sg was set to 1 and size to 600, yielding the most accurate similarity calculation results. Table 4 presents some of the word vector training results when using the Skip-gram algorithm with a dimensionality of 600. Table 4 illustrates the word vector representation results for five words using a heatmap, where the horizontal axis represents the dimensions of the word vectors, and the vertical axis represents the words. The depth of the color fill reflects the magnitude of the vector dimensions. The visualization of word vectors through heatmaps allows analysis of the distribution and characteristics of word vectors. As shown in Figure 4, the results indicate significant differences across dimensions for the five words. The vector representation results overcome the deficiencies of traditional methods by offering dense representations that enable the differentiation of words through high-dimensional features.

3.4. Similarity calculation results and analysis

Based on the word vector training results, similarity calculations were conducted using cosine similarity. Then, the top n indicators most similar to the target indicator were identified, and an interactive program was used to find and display them. The similarity calculation results range from 1 to -1, with values closer to 1 indicating higher similarity between two sentences, and values closer to -1 indicating lower similarity. Table 5 presents a random selection of six indicators, along with the top two indicators with the highest similarity results with these six indicators and their similarity calculation results. From the table, the similarity calculation results provide reference values for the selection of substitute indicators. An interactive program was used to randomly input an indicator and visually output the top five indicators with the highest similarity scores to that indicator; the output results are displayed in a bar chart. As shown in Figure 5, upon entering "Average Daily Concentration of Sulfur Dioxide", the corresponding bar chart was generated.

From Figure 5, we can see that the top five indicators most similar to the "Daily Average Concentration of Sulfur Dioxide" indicator are, in order of similarity, Annual Average Concentration of Sulfur Dioxide, Sulfur Dioxide Concentration and Nitrogen Dioxide Concentration, Daily Maximum 8-hour Average Concentration of Ozone, Urban Area PM, Annual Average Concentration, and Annual Average Concentration of PM2.5, with similarity scores of 0.993, 0.992, 0.990, 0.989, and 0.989, respectively. This means that in the process of assessing a city, if the indicator "Daily Average Concentration of Sulfur Dioxide" is missing or unavailable, these indicators can provide reference for urban sustainable development assessment.

4. Conclusion

This study collected 1 238 urban sustainable development assessment indicators to address the issue of missing or unavailable indicators for urban sustainable development assessment. The similarity between indicators was calculated by combining the Word2Vec and cosine similarity techniques. The effects of different algorithm models and parameter settings on the similarity calculations were compared using the accuracy, precision, recall, and F1 score. The results show that the model has the best performance when sg is set to 1 and size to 600, that is, when the Skip-gram algorithm has a word vector dimensionality of 600. Using these parameters to train the Word2Vec model and then applying the training results to the cosine similarity algorithm, the similarities between the indicators were obtained. Finally, the similarity results between the indicators were visually pre- sented, demonstrating the effect of seeking substitute indicators. The results of this study offer feasible methods for selecting substitute indicators for future urban sustainable development assessment. However, this study did not compare the different text vectorization methods. Future research should include comparative studies of different models to explore improvement schemes to enhance the effectiveness of selecting substitute indicators for sustainable urban development in China.

Acknowledgments

Corresponding author Tianshu Yu is a postdoctoral researcher jointly trained by China Architecture Design & Research Group and Nankai University.

Funding

The paper are supported by the National Key Research and Development Program of China under the theme "Key technologies for urban sustainable development evaluation and decision-making support" [Grant No. 2022YFC3802900] and the Guangxi Key Research and Development Program [Grant No. Guike AB21220057].

Sidebar

Received 02 February 2024; Accepted 16 July 2024

Footnote

* Corresponding author.

E-mail address: [email protected] (T. Yu)

References

References

Abraham, D. B., 2021. Local SDG indicators for US cities and communities. In Promoting the sustainable development goals in north American cities: Case studies 8% best practices in the science of sustainability indicators, 147-155.

Adams, B., Judd, K, 2016. 2030 Agenda and the SDGs: indicator framework, monitoring and reporting. https://www. globalpolicywatch. org/blog/2016/03/18/2030agenda-sdgs-indicator/.

Angelidou, M., Psaltoglou, A., Komninos, N., Kakderi, C., Tsarchopoulos, P., Panori, A., 2018. Enhancing sustainable urban development through smart city applications. Journal of Science and Technology Policy Management 9, 146-169.

Brunn, S. D., Williams, J. F., Zeigler, D. J, 2003. Cities of the world: world regional urban development. Washington D.C.: Rowman & Littlefield Publisher.

Chowdhary, K.R., 2020. Natural language processing. Fundamentals of Artificial Intelligence. New Delhi: Springer India, 603-649.

Church, K. W., 2017. Word2Vec. Natural Language Engineering, 23(1), 155-162.

Gao, F., Zhao, X.Y., Song, X.Y., Wang, B., Wang, P.L., Niu, Y.B., Wang, W.J., Huang, C. L., 2019. Connotation and evaluation index system of beautiful China for SDGs, Advances in Earth Science, 34(3): 295-305.

Hanley, N., Moffatt, 1., Faichney, R., Wilson, M., 1999. Measuring sustainability: a time series of alternative indicators for Scotland. Ecological Economics 28, 55-73.

Harispe, S., Ranwez, S., Janaqi, S., Montmain, J., 2015. Semantic Similarity from Natural Language and Ontology Analysis. Cham: Springer International Publishing.

Huan, Y.Z., Wang, L.Q., Burgman, M., Li, H.T., Yu, Y.R., Zhang, J.P., Liang, T., 2022. A multi-perspective composite assessment framework for prioritizing targets of sustainable development goals. Sustainable Development 30, 833-847.

Lee, B.X., Kjaerulf, F., Turner, S., Cohen, L., Donnelly, P.D., Muggah, R., Davis, R., Realini, A., Kieselbach, B., MacGregor, L.S., Waller, I., Gordon, R., Moloney-Kitts, M., Lee, G., Gilligan, J., 2016. Transforming our world: implementing the 2030 agenda through sustainable development goal indicators. Journal of Public Health Policy 37, 13-31.

Li, F., Liu, X.S., Hu, D., Wang, R.S., Yang, W.R., Li, D., Zhao, D., 2009. Measurement indicators and an evaluation approach for assessing urban sustainable development: a case study for China's Jining City. Landscape and Urban Planning 90, 134-142.

Lilleberg, J., Zhu, Y., Zhang, Y.Q., 2015. Support vector machines and Word2Vec for text classification with semantic features. 2015 IEEE 14th International Conference on Cognitive Informatics 8% Cognitive Computing. 2015, Beijing, China. IEEE, 136-140.

Lind, N., 2019. A development of the human development index. Social Indicators Research 146, 409-423.

Lynch, A., LoPresti, A., Fox, C., 2019. The 2019 US Cities Sustainable Development Report. New York: Sustainable Development Solutions Network (SDSN).

Marvuglia, A., Havinga, L., Heidrich, O., Fonseca, J., Gaitani, N., Reckien, D., 2020. Advances and challenges in assessing urban sustainability: an advanced bibliometric review. Renewable and Sustainable Energy Reviews 124, 109788.

Nadkarni, P.M., Ohno-Machado, L., Chapman, W.W., 2011. Natural language processing: an introduction. Journal of the American Medical Informatics Association 18, 544-551.

Nourry, M., 2008. Measuring sustainable development: Some empirical evidence for France from eight alternative indicators. Ecological Economics 67, 441-456.

Rong, X., 2014. word2vec parameter learning explained.: 1411.2738.https://arxiv.org/ abs/1411.2738v4

Salem, M., Tsurusaki, N., Divigalpitiya, P., Osman, T., Hamdy, O., Kenawy, E., 2020. Assessing progress towards sustainable development in the urban periphery: a case of greater Cairo, Egypt. International Journal of Sustainable Development and Planning 15, 971-982.

Satterthwaite, D., 2011. How urban societies can adapt to resource shortage and climate change. Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences 369, 1762-1783.

Shulla, K., Leal-Filho, W., 2023. Achieving the UN Agenda 2030: overall actions for the successful implementation of the sustainable development goals before and after the 2030 deadline.

Xia, P.P., Zhang, L., Li, F.Z., 2015. Learning similarity with cosine similarity ensemble. Information Sciences 307, 39-52.

Xiong, Z.Y., Shen, Q.Q., Xiong, Y.S., YijieWang, Li, W.Z., 2019. New generation model of word vector representation based on CBOW or skip-gram. Computers, Materials & Continua 60, 259-273.

Yacouby, R., Axman, D., 2020. Probabilistic Extension of Precision, Recall, and F1 Score for More Thorough Evaluation of Classification ModelsProceedings of the First Workshop on Evaluation and Comparison of NLP Systems. Online. Stroudsburg, PA, USA: Association for Computational Linguistics, 79-91.

Zhang, F., Wang, Y., Ma, X.J., Wang, Y., Yang, G.C., Zhu, L., 2019. Evaluation of resources and environmental carrying capacity of 36 large cities in China based on a support-pressure coupling mechanism. Science of the Total Environment 688, 838-854.

Zhang, X., Shao, C., Zhou, L., 2022. Blue book on urban human settlements in China. Beijing: China City Press.

Zhu, J., Sun, X.Z., He, Z., Zhao, M.Y., 2019. Are SDGs suitable for China's sustainable development assessment? An application and amendment of the SDGs Indicators in China. Chinese Journal of Population Resources and Environment 17, 25-38.