Application Framework of Multi-Criteria Methods in Sustainability Assessment

Abstract

In the contemporary literature on sustainability, one can notice an increasingly frequent use of Multi-Criteria Decision Analysis (MCDA) methods instead of classic sustainability indices. The MCDA method should be tailored to the specific sustainability problem and decision situation so that its results are credible and satisfactory for the decision-maker. Therefore, the following research questions arise: (1) which MCDA methods are most often used in sustainability problems, and (2) which methods should be used depending on the characteristics of a particular sustainability decision problem and its assessment. The aim of the article is to scientifically analyse the applicability of various MCDA methods in decision-making problems related to sustainability, sustainable development, and sustainability assessment. In the article, based on the analysis of the literature, a set of features has been developed that determines the possibility of using individual MCDA methods in sustainability problems. Then, the characteristics of 28 methods are presented and the framework for selecting the MCDA method for the purpose of sustainability decision problems is indicated. As a result of the conducted research, it was found that the most commonly used MCDA methods in sustainability problems are primarily methods based on arithmetic aggregation of criteria. In addition, fuzzy methods and fuzzy modifications of classical methods are used more often. Research has established that MCDA methods are more functional than classic sustainability indices. In addition, the use of MCDA methods in the assessment of sustainability gives much more flexibility than the use of classic indices. The proposed framework allows the decision-maker to independently assess the potential of using individual multi-criteria methods in specific decision-making problems related to sustainability. The framework enables the selection of an appropriate MCDA method depending on the defined needs of the decision-maker, resulting from the decision problem, its structure, and decision-making situation.

1. Introduction

Sustainability assessment is the subject of numerous scientific studies. Usually, the so-called sustainability indices [1] allow for the quantification of individual activities and the measurement of sustainable development and sustainability on a local and global scale. These indices are usually based on simple calculation procedures, consisting of an arithmetic or geometric means. The indices used usually do not allow to capture the uncertainty of data and uncertainty of assessments in any way, and also do not provide the opportunity to perform a sensitivity analysis of the solution depending on changes in the input data. It should also be noted that the indices are adapted to the implementation of specific tasks, and therefore they are characterized by very little flexibility, not allowing the choice of the applied sustainability indicators.

The literature emphasizes the wide applicability of methods derived from operations research and management science in the problems of sustainable development [2,3]. This is because operations research is, in many respects, similar to sustainability research. These similarities include: similar groups of recipients; a wide range of methodological approaches used; taking into account many, often contradictory perspectives; interdisciplinarity; and applications in the planning and implementation of activities [4]. One of the dynamically developing trends in operations research, that is, Multi-Criteria Decision Analysis (MCDA), occupies a special place in the assessment of sustainability. MCDA methods are designed to solve complex decision problems with many conflicting criteria and goals and sources of uncertainty [5,6]. Meanwhile, such features usually characterize decision problems related to sustainable development and sustainability assessment [3,4]. In particular, it is recognized that the assessment of sustainability is in fact a multi-criteria problem, as it consists in seeking a compromise between contradictory or conflicting indicators [7]. Moreover, the characteristics of MCDA methods are consistent with the concepts of strong and weak sustainability, due to the different degrees of criteria compensation used in each method. The concept of strong sustainability is reflected by a low degree of compensation, and weak sustainability corresponds to a high degree of criteria compensation [8]. It should also be noted that MCDA methods allow to avoid the problematic monetization of individual dimensions of sustainability, that is, expressing them in the form of monetary values [6]. For the above reasons, MCDA methods are increasingly used in the assessment of sustainability [7,9,10,11]. In addition, it should be noted that in practice, even in the simplest indices of sustainability, a multi-criteria methodology is used. This is due to the fact that both the arithmetic and geometric mean form the basis of more advanced methodologies used in MCDA methods [12]. Moreover, the MCDA methods can be treated as a subset of the sustainability assessment methods, allowing for a better understanding of the assessment results and a more precise determination of the degree of achievement of individual sustainability objectives [13].

As suggested by Cinelli et al. [3], the MCDA method should be tailored to a specific problem and decision situation. This means that there is no one-size-fits-all method that can be applied to all sustainability decision problems. On the other hand, it should be pointed out that most often, the choice of a multi-criteria method is not based on its suitability to the problem, but results only from the knowledge of the method by the decision-maker or the availability of software using a given method [14]. On the other hand, the large variety of multi-criteria methods used in sustainability problems gives the decision-maker the opportunity to choose the method with the features best suited to the problem itself. Sometimes it is also worth using many methods and comparing their results in order to choose the best solution [15]. Therefore, the following research questions arise: (1) which MCDA methods are most often used in sustainability problems, and (2) which methods should be used depending on the characteristics of a particular sustainability decision problem and its assessment.

The aim of the article is to scientifically analyse the applicability of various MCDA methods in decision-making problems related to sustainability, sustainable development and sustainability assessment. The aim implies the contribution of the article, which is the framework for selecting the MCDA method for the purpose of sustainability decision problems. The implementation of the set goal began with a literature review on the applicability of MCDA methods in various decision-making problems. This review is presented in Section 2. As a result of the conducted review, a set of features was developed that determines the applicability of individual MCDA methods to sustainability problems. The defined set of features is discussed in detail in Section 3. Section 4 presents the characteristics of the MCDA methods most commonly used in decision-making problems related to sustainability, sustainable development, and sustainability assessment. These characteristics were developed on the basis of the set of characteristics presented in Section 3. The article ends with the discussion and conclusion presented in Section 5 and Section 6.

2. Literature Review

In the literature, you can find many works on the applicability of individual multi-criteria methods in decision problems depending on the characteristics of the problem. These are both works examining the general specificity of decision problems, as well as directly related to the decision problems of sustainability. The analyses presented in them are based primarily on the characteristics of the problem, and the selection of a specific method depends on whether its features are consistent with the characteristics of the problem. Due to the number of publications on this subject, this section presents only selected approaches to the study of the applicability of multi-criteria methods.

Among the works dealing with the problem of the possibility of using the MCDA methods in various types of general decision-making problems, there are, among others, publications such as that of Roy and Słowiński [16] and Wątróbski et al. [17]. On the other hand, among the works relating to the applicability of multi-criteria methods in sustainability problems, there are, among others, articles by Sadok et al. [18], Rowley et al. [8], De Montis et al. [19], Moghaddam et al. [20], Polatidis et al. [21], and Cinelli et al. [3].

Roy and Słowiński [16], as the basic issue determining the applicability of individual methods in various decision-making contexts, indicated the type of solution that the decision-maker wants to obtain (among others, numerical values of alternatives, full or partial ranking, a subset of alternatives, assigning alternatives to different categories). Moreover, the authors point out that the applicability of multi-criteria methods also depends on the key characteristics of the decision problem, relating to: the type of performance scale; the ease of obtaining the preferential information required by the method; taking into account the uncertainty, imprecision, and vagueness of data in the problem; acceptability of compensation between the criteria; and taking into account the interaction between the criteria. Roy and Słowiński also formulated additional questions determining the applicability of the method: whether the method is sufficiently understandable for the stakeholders; whether there is an axiomatic characteristic of the method available and is it acceptable in the considered decision context; and whether weaknesses of the method could impact the final choice.

Wątróbski et al. [17] indicated four basic aspects related to the selection of a method to solve a specific decision problem. These are general characteristics relating to: the consideration and type of criterion weights; the type of comparison scale of alternatives; the consideration and type of uncertainty in the problem; and decision-making problematics, including the type of ranking (full or partial). Within the framework of individual characteristics, sub-characteristics were also distinguished, detailing the description of the problem depending on the information possessed by the decision-maker. The authors emphasize that both the weights of the criteria and the scales of comparisons can be of a qualitative, quantitative, or relative nature. Uncertainty can relate to the input data (weighting of the criteria and the performance of alternatives) and to the preferences of the decision-maker (indifference and preference thresholds). Analysis of the applicability of MCDA methods in a decision problem can be carried out on the basis of method characteristics, rules, and a decision tree.

Sadok et al. [18] considered the applicability of multi-criteria methods in problems related to sustainable agricultural systems. They pointed out that, in problems of sustainability, continuous multi-objective methods are usually not applicable. This is because ex ante sustainability assessments tend to include qualitative assessments and data gaps and inconsistencies to which continuous methods are particularly sensitive. In turn, they consider the applicability of discrete methods through the prism of the commensurability, compensation, and comparability of criteria, as well as the method’s ability to deal with mixed (quantitative and qualitative) scales of criteria. The authors express the opinion that the method used in the assessment of sustainability cannot enforce commensurability and full comparability of the criteria, and should be characterized by the lack of compensation and allow for quantitative and qualitative data to be taken into account. They ignored the method’s ability to deal with uncertainty because, in their opinion, almost all multi-criteria methods can be linked to the uncertainty-handling procedure.

Rowley et al. [8] defined a decision diagram that allows to choose the appropriate multi-criteria method depending on the specificity of the decision problem related to sustainability. This diagram takes into account such characteristics of the problem and methods as: the type of decision-maker (individual or group), the type of criteria (simple or complex), the application of the veto threshold (in the sense of the criterion value that causes rejection of a given alternative), the application and type of criteria weights (cardinal or ordinal), hierarchy of the decision problem, and the used model of preference aggregation. However, the discussed diagram contains some inaccuracies; for example, the application of the veto threshold excludes the possibility of determining the weights of the criteria. However, like the other approaches mentioned in this section, it shows some useful characteristics of the decision problem in order to select the appropriate multi-criteria method.

De Montis et al. [19] analysed the applicability of multi-criteria methods in the problems of sustainable development, taking into account three groups of characteristics: operational components of the methods, as well as the applicability of the methods due to the user’s context and the structure of the decision problem. Each group contains a relatively large number of characteristics, but the authors emphasize that without reference to the specific characteristics of the problem for which the method will be applied, it is not possible to choose the best method. Therefore, according to the authors, the most important in general sustainability problems are the characteristics related to:

• The ability to deal with complex decision problems (dependencies between criteria, completeness of criteria, nonlinearity of preferences, applicability of various geographic and institutional scales, ability of the method to examine social and technical issues, taking into account quantitative and qualitative data, as well as risk and uncertainty);

• No substitution (and therefore no criteria compensation, which is tantamount to strong sustainability;

• The ability to include multiple decision-makers (also including transparency of the decision-making process and support for communication between stakeholders);

• Informing stakeholders in order to increase their knowledge and enable the development of a compromise (including supporting the process of problem structuring, supporting methodology, transparency of the decision-making process, and type of criteria weights).

Moghaddam et al. [20] considered the usefulness of multi-criteria methods in the problems of sustainable energy planning. According to the authors, the multi-criteria method used in this type of problem should consider the problematic of ranking. In addition, it should take mixed data (qualitative and quantitative) into account, using the distance function to compare alternatives. The selected method should ensure full comparability of alternatives, and thus also allow to obtain a complete ranking without an incomparability relation. At the same time, the method should be flexible enough to allow the use of various preference functions (criteria functions) and support individual and group decisions. It should ensure a low degree of criteria compensation, but should not be completely non-compensatory. In addition, the authors suggest that it allows to determine the weights of the criteria using pairwise comparisons, as is the case with the AHP method. The optimal method to be used should also not be too complicated, so that its computational apparatus does not exceed the knowledge of the decision-maker, and at the same time it cannot be computationally trivial. Finally, software implementing the chosen method should be available, and the literature analysis should show that the method is actually applied to the class of decision problems.

Like Moghaddam et al. [20], as well as Polatidis et al. [21] considered the problem of the applicability of multi-criteria methods in problems related to energy planning, taking into account the aspects of sustainable development. They distinguished between five groups of characteristics that determine the applicability of multi-criteria methods in the problems of energy sustainability. These are: modelling the preferences of the decision-maker, the strength of sustainability, theoretical and technical aspects of methods, the approach to uncertainty, and practical requirements. In each group, requirements for more detailed characteristics of each method are indicated. In the authors’ opinion, criteria weights used to model preferences should be used in terms of importance coefficients, not trade-offs. In addition, the method should have a degree of criteria compensation that is as low as possible to enable a strong level of sustainability to be obtained. In terms of technical aspects, the authors suggest that multi-criteria methods should enable the use of quantitative and qualitative data, allow for the construction of a hierarchy of criteria, and give some flexibility to define the decision problem through the use of various configuration parameters (interaction with the method). Additionally, the method should enable uncertainty to be captured in the broadest possible range. The authors emphasize that the probabilistic approach, derived from classical decision analysis and possible to be applied in methods based on the utility theory, is insufficient here. The methods using the fuzzy set theory and the methods based on outranking relations are more effective in this respect. Regarding the practical aspects of each method, the following are important: ease of use of the method; the possibility of considering a large number of decision-makers and a large number of criteria and alternatives; the possibility of using many different parameters related to the capture of inaccurate and uncertain criteria; ease of interpretation of particular parameters of the method; and low time and financial requirements related to the use of the method.

Cinelli et al. [3] considered the potential of using multi-criteria methods in the assessment of sustainability. The characteristics that determine the applicability of a given method to test the sustainability have been divided by the authors into three groups: the scientific aspects of the method (relating to the input data and the calculation procedure), feasibility, and usability. For each group, they indicated the desired characteristics of the method to be used in the sustainability assessment. Within the scientific aspects related to input data, these are: the ability to take quantitative and qualitative data into account, and the ability to take a life-cycle perspective in the assessment into account. On the other hand, among the scientific aspects related to the calculation procedure, the authors indicated: the use of weights in the sense of importance coefficients, not trade-offs, the possibility of using thresholds (indifference, preference, veto), no compensation of criteria, taking uncertainty and the possibility of conducting a sensitivity analysis into account, the method’s resistance to the occurrence of the phenomenon of ranking inversion after adding a new alternative. In turn, in terms of feasibility, the desirable characteristics are: software support of the method and ease of use of the method. Usability directly relates to the ease of understanding of a method by a decision-maker. The authors state that multi-criteria methods have a high potential for applications in the assessment of sustainability. Moreover, despite identifying the desirable characteristics for the sustainability assessment, they note that the selection of the appropriate method must be based on both knowledge of the method and knowledge of the decision problem to be investigated.

On the basis of the discussed publications, a set of discrete features of multi-criteria methods was developed, conditioning the possibility of using individual methods in problems related to sustainability. These are the features most frequently indicated in the works cited.

3. Features of Multi-Criteria Methods Deciding Their Application in the Sustainability Research

The presented set of features was prepared by taking into account the fact that in individual publications, some features, despite the fact that they refer to the same aspects of the method, were named differently. For example, the type of the obtained solution indicated by Roy and Słowiński [16] can be identified with the problematic of decision, mentioned as one of the basic features in the publications of Wątróbski et al. [17] and also Moghaddam et al. [20] and Cinelli et al. [3], who specifically indicated the problematic of the ranking. The developed set of features of multi-criteria methods includes:

• The problematic of the decision under consideration;

• Applied preference relations;

• The order of alternatives obtained;

• Alternative performance scales used;

• Type of criteria weights;

• Meaning of criteria weights (interpretation);

• Degree of criteria compensation;

• Recognition of uncertainty;

• Taking into account the relationship between the criteria;

• Ease of use of the method;

• Supported number of decision-makers; and

• Software support.

Table 1 shows the relationship of individual features with the review publications cited in Section 2, in which they were indicated as important features when choosing the multi-criteria method.

Multi-criteria methods are designed to solve various decision problems. In fact, the following problematics are distinguished [22]:

• Choice (P.α)—the method helps the decision-maker to select a subset of alternatives that is as small as possible, from which the decision-maker can possibly choose a single alternative;

• Sorting (P.β)—the method supports the decision-maker by assigning each alternative to categories that are previously defined by the decision-maker;

• Ranking (P.γ)—the method supports the decision-maker by determining the order of alternatives, which is obtained by placing alternatives in equivalence classes, fully or partially ordered according to preference;

• Description (P.δ)—the method supports the decision-maker in describing alternatives and their consequences.

Moreover, Ishizaka and Nemery [23] also indicated the following problematics:

• Elimination (P.ε)—this is a special kind of sorting problem;

• Design (P.ζ)—the method identifies or constructs a new alternative that meets the goals and aspirations of the decision-maker (this problematic is related to continuous, rather than discrete, multi-criteria methods);

• Elicitation (P.η)—the method is to indicate to another multi-criteria method the parameters of preferences or information of a subjective nature (this problematic is related to some interactive methods).

Most of the multi-criteria methods consider the problematic of ranking, and only a few take up the remaining problematics.

In multi-criteria methods, the order between alternatives is expressed by the preference relation (I—indifference, P—strict preference, Q—weak preference, R—incomparability, S—outranking, I/R—non-preference, Q/P—preference, J—J-preference, K—K-preference) [24,25]. These relations depend on the applied thresholds. Methods based on a single synthesizing criterion use the I and P relations, and methods based on the outranking relation may use more preference relations, mainly Q, R, and S. Relations can refer to criteria comparisons, as well as intermediate stages of the calculation procedure and final ranking. It should also be noted that the preference relations used in a given method have an impact on the ordering of alternatives and are related to the way in which uncertainty is perceived by the multi-criteria method.

The method of ordering the alternatives is closely related to the incomparability relation R. Namely, the multi-criteria method, which does not take into account the incomparability relation, usually allows to obtain a total order, that is, full comparability of all alternatives in the problem of ranking P.γ. On the other hand, taking into account the incomparability relation by the method results in obtaining a partial order, meaning that there may be alternatives that cannot be compared with others in the ranking [26]. The methods examining the problem of sorting P.β may, in turn, generate a full or partial interval order [27]. Moreover, some methods using the incomparability relation generate a graph core, or a subset of alternatives, considering the problem of choosing P.α [28].

Both the performance of alternatives and the weighting of criteria can be expressed on various scales, depending on the nature of the data [29]. The most common scales are qualitative and quantitative, while Roy [30] indicates that they can be identified with ordinal and cardinal scales, respectively. The values on the qualitative scales can be described verbally (linguistically) or numerically; however, in both cases the intervals between the successive values on the scale may be different. This means that the pairs of consecutive values on the scale reflect different preference differences. On the quantitative scales, on the other hand, the values are described numerically and the steps of the scale are defined by reference to a clearly defined reference unit defining the difference between two adjacent scale values. The quantitative scale is undoubtedly the quotient scale, and the interval scale is defined as intermediate between qualitative and quantitative. The type of performance scale determines the acceptable method of data normalization in a given method. The AHP and ANP methods are specific in terms of the scale used. Although they usually use a relative qualitative scale [1,2,3,4,5,6,7,8,9], a quantitative scale and natural criteria values can also be used [31]. It should also be noted that the method of capturing more complex quantitative information also allows to capture simpler qualitative information, such as the propagation rules presented by Guitouni et al. [32]. On the other hand, for the entry of a method that allows only the use of a qualitative data, quantitative data cannot be provided without loss of information.

Criteria weights can be interpreted differently, depending on the multi-criteria method used. If weights are used in the sense of trade-offs or the intensity of preferences, then they define the rate of substitution [33]. This means that having two criteria and certain criteria performance of alternatives, reducing the performance of one of the criteria by a unit value and increasing the performance of the other criteria by the ratio of the weighting of the criteria, makes the alternative before and after modification indistinguishable [34]. Due to the occurrence of substitution, it is obvious that the use of weights in the sense of trade-offs makes the method meets the weak sustainability paradigm. Nevertheless, Roy [34] points out that determining the marginal rate of substitution may be useful in the decision-making process, especially when there is no dominance. Determining the rate of substitution on the basis of weights can facilitate and accelerate reaching a compromise between stakeholders. Substitution does not occur in the case of weights used in the sense of importance coefficients or expressed on an ordinal scale [33]. Therefore, multi-criteria methods using weights in this sense are characterized by stronger sustainability.

The meaning or the method of interpreting the weights is essential for determining the degree of criteria compensation in the method [35]. The very phenomenon of criteria compensation is that the loss of a given alternative on one of the criteria is compensated by its profit on another criterion [36]. The literature distinguishes multi-criteria methods: compensatory, partially compensatory, and non-compensatory [6,27]. It is generally accepted that the methods using a single synthesizing criterion are more compensatory than the methods based on the outranking relation [8]. Moreover, the methods using the additive aggregation model are more compensatory than the methods using the multiplicative model [33,37]. Munda [35] points out that compensation is related to the method of interpreting criteria weights. According to him, the use of weights in the sense of trade-offs between criteria makes the method compensatory. On the other hand, the use of weights in the sense of importance coefficients makes the method non-compensatory. However, with some methods, such as the Preference Ranking Organization METHod for Enrichment Evaluation (PROMETHEE), it is difficult to determine whether weights are used as trade-offs or importance coefficients. This causes some problems with determining the degree of compensation of these methods [3]. Similarly, with other methods, it is often difficult to determine the degree of compensation. For example, methods from the ELimination Et Choix Traduisant la REalité (ELECTRE) family are considered non-compensatory by many researchers [28,38]. However, others consider that they are partially compensated [20,27]. Hence, it is not easy to classify the MCDA method as compensatory, non-compensatory, or partially compensatory. In practice, the degree of method compensation depends on the parameters of the decision problem provided by the decision-maker. For example, in the PROMETHEE and ELECTRE methods, the degree of compensation can be adjusted to some extent by appropriately manipulating the values of the indifference (q) and preference (p) thresholds [3,21]. Additionally, in the ELECTRE family methods, compensation can be limited by using appropriate veto threshold (v) values for criteria [3,36]. In turn, in the Dominance-based Rough Sets Approach (DRSA) method [39], which is considered non-compensatory [3,16], the compensation in practice is regulated by the attributes of objects/alternatives defined by the decision-maker in the data table. Note that compensation confirms substitution, so compensatory methods are applicable to problems related to weak sustainability, and non-compensation methods are appropriate to problems related to strong sustainability [3,8].

Many multi-criteria decision problems are characterized by uncertainty and imprecision. This uncertainty may concern: input data representing potential alternatives and criteria weights, or the preferences of the decision-maker when comparing alternatives [40]. The first type of uncertainty results from the fact that at the time of making a decision, the assessment of potential alternatives is usually ex ante, so the decision-maker is not able to completely accurately and reliably estimate the weighting of the criteria and the consequences of each of the alternative. The first type of uncertainty is captured by fuzzy multi-criteria methods (using the fuzzy set theory) [41], most often based on a single synthesizing criterion and using the relations I and P [30]. The second type of uncertainty is when the decision-maker cannot decide with certainty that one of the potential alternatives is better than the other in terms of a certain criterion. To solve decision problems in which there is the second type of uncertainty, multi-criteria methods based on the outranking relation are used, often using the fuzzy preferences and relations I, P, Q, R, S [42]. The literature also uses fuzzy methods using the outranking relation, which take into account both types of uncertainty and may take into account additional relations, such as Q/P and J [43].

Making decisions based on many interdependent criteria is the norm for standard business decisions [44]. Nevertheless, most multi-criteria methods assume independence between criteria [45], and therefore it is difficult to apply them to complex decision problems in which such dependencies occur [19]. As Gölcük and Baykasoğlu [46] point out, the assumption of the independence of criteria is not realistic in many real decision-making problems. On the other hand, the omission of the existing dependencies means that the problem is not correctly reflected in the decision model [47], as a result of which a multi-criteria solution is not as useful as it could be [44]. In extreme cases, this leads to wrong results and may lead to wrong decisions [48]. Moreover, even though we strive to remove the dependent criteria, there is still no guarantee that the criteria are completely independent of each other [45]. The relationships between the criteria can be understood as preferential, additive, and utility-related dependencies [19]. Dependencies are of particular importance in the case of methods based on a single synthesizing criterion. Such methods should allow modelling the aforementioned dependencies, otherwise only independent criteria should be taken into account in the decision problem. In the case of methods based on the outranking relation, independence of criteria is recommended, but it is not a restrictive requirement. Nevertheless, it is assumed that the relationships between the criteria affect the final weighting of the criteria [48]. Among the dependency models, first of all, the hierarchical and network model should be mentioned. In the hierarchical model, individual elements of the hierarchy (including criteria and alternatives) are not related to all others, but are grouped into disjoint subsets. If there is a relationship between certain criteria, they should be placed at the same level of the hierarchy and linked to the same parent [19]. Thanks to this, the decision-maker, when specifying the weights of the criteria, considers only a limited subset of them at one time. This approach makes it easier to determine the weights of criteria while taking into account their interdependencies. It should be noted, however, that the hierarchical model, as the name suggests, allows to model only hierarchical relationships between criteria, and the remaining relationships must be taken into account by the decision-maker implicitly. Therefore, this model is not sufficient to fully define the decision problem in which there are dependencies other than hierarchical [48]. On the other hand, the network model, apart from hierarchical relationships, also enables modelling of feedback as well as internal and external (in relation to a given subset of criteria) dependencies between criteria at the same level of hierarchy [49]. The hierarchical model is used primarily in the Analytic Hierarchy Process (AHP) method, as well as in hierarchical modifications of other methods. On the other hand, the network model is used in the Analytic Network Process (ANP) and Decision-Making Trial and Evaluation Laboratory (DEMATEL) methods.

The ease of use of the method means that the method is easier to apply than others [50], but it is also related to the understanding by the decision-maker [51]. Ease of use is determined by the number and type of parameters required by the method. Some multi-criteria methods, such as the Multiple-Attribute Utility Theory/Multiple-Attribute Value Theory (MAUT/MAVT), require a time-consuming assessment of the utility function, and the use of AHP/ANP methods involves the necessity to carry out numerous and time-consuming pairwise comparisons. Moreover, the use of many methods based on a single synthesizing criterion is connected with the necessity to identify trade-offs between the criteria. On the other hand, the use of methods based on the outranking relation requires the identification of thresholds included in these methods, and in the case of the PROMETHEE method family, it is additionally necessary to indicate the appropriate preference function (criterion function) [3]. However, for interactive methods such as DRSA, examples of alternatives and their aggregated overall assessments must be provided. Referring to the ease of understanding of the method by the decision-maker, it should be noted that decision-makers as well as other stakeholders have a need to understand the method used [16], and they are usually not experts in the field of multi-criteria methods [21]. Therefore, their understanding of the mathematical apparatus of a given method may be limited [21], as a result of which they cannot delve into the decision-making process [20]. As a result, they may treat a given method as a “black box” and distrust its results [20,51], and even feel manipulated by the results of the method used [21]. It is recognized that in such a situation, it is not worth using the method [51], or it is necessary to spend a sufficient amount of time explaining the relationship between the input data and the results of the method, without going into the calculation details [16]. According to various researchers, one of the easier to understand multi-criteria methods is the PROMETHEE method, considered less complicated than the ELECTRE or even AHP methods [21,50,51], although the simplest one is Weighted Arithmetic Mean/Simple Additive Weighting (WAM/SAW) [52]. On the other hand, when it comes to representing the results of the method, some researchers believe that the results of the methods using the outranking relation are easier to understand than the results of the methods based on the single synthesizing criterion [51].

The high complexity of the socio-economic environment often makes it difficult for a single decision-maker to take into account all important aspects of some decision problems [53]. Therefore, important decisions in organizations are usually made by a group of people whose collective recommendations are considered more pragmatic than individual opinions [54]. Group decisions can be made on the basis of three basic models of group decision procedures [55]:

• Sharing—the decision is made by consensus (e.g., the values of alternatives and criteria weights are jointly established, and then the decision problem is solved based on a method designed for a single decision-maker);

• Aggregation—the decision is made on the basis of a compromise solution (e.g., each of the decision-makers generates their own ranking of alternatives, and the rankings obtained in this way are then aggregated into a group ranking);

• Comparison—the decision is made on the basis of a consensus based on the negotiation of independent individual results (e.g., each decision-maker generates his own ranking of alternatives and then jointly selects one of these rankings).

All multi-criteria methods support sharing and comparison models, but only some methods allow for the aggregation of preferential information from multiple decision-makers or reflecting different priorities or scenarios. These types of methods include: AHP, ANP, PROMETHEE Group Decision Support System (GDSS), and the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS). On the other hand, it should be noted that for methods based on a single synthesizing criterion, it is relatively easy to add the step of aggregating individual scores to the group score. Development of this type of method into a group form can be reduced to the calculation of the arithmetic or geometric mean of individual assessments, as in the case in the group versions of the methods: AHP, ANP, or TOPSIS [56,57]. Similarly, in the fuzzy methods, at the preliminary stage, the aggregation of individual criteria assessments into fuzzy group scores may be performed, as is the case, for example, in the Fuzzy TOPSIS (FTOPSIS) method [58].

The last considered feature of multi-criteria methods is the availability of software supporting a given method. This is an important issue because the selection of a multi-criteria method for a specific problem is often motivated by the availability of software supporting a given method [14]. Moreover, decision-makers usually choose well-known software supporting the decision-making process, and thus also choose a specific multi-criteria method [59]. For many methods used in sustainability problems, software is available to support the work of the decision-maker. These are, among others, the following software: Super Decisions (AHP/ANP), Right Choice (WAM/SAW and the additive version MAUT/MAVT), Visual PROMETHEE (PROMETHEE), SDI Tools (TOPSIS), J-Electre (ELECTRE), Win4 DEAP (DEA—Data Envelopment Analysis), Naiade (NAIADE—Novel Approach to Imprecise Assessment and Decision Environments), ruleRank (DRSA). Some of the software offer extensive support for the decision-making process, enabling, for example, sensitivity analysis and other analyses that increase the transparency of the solution, while others generate only a ranking or a subset of recommended alternatives. It should also be noted that some methods are implemented as tools available only as web applications.

4. Characteristics of MCDA Methods in the Context of Sustainability Problems

According to Moghaddam et al. [20], one of the basic conditions confirming the possibility of applying the multi-criteria method is the fact of its actual use in a given class of decision problems. Therefore, about the applicability of multi-criteria methods in the decision-making problems of sustainability, the most talked about are reviews, which provide information on the methods most often used to solve these types of problems. Among such works, there are publications relating to: general sustainability problems [7,60], assessment of the sustainability of infrastructure investments [61], assessment of the sustainable life cycle of products [62] and production technologies [10], sustainable supply chain management [63], and sustainable energy planning and management [64,65,66]. The percentage share of individual multi-criteria methods in solving problems related to sustainability, determined on the basis of the cited publications, is presented in Table 2. It should be noted that this table includes methods that have been mentioned in more than one review publication.

The analysis of Table 2 shows that the most frequently used multi-criteria methods based on a single synthesizing criterion are: AHP and WAM/SAW. The following methods are used less frequently: TOPSIS, ANP, MAUT/MAVT, VIšekriterijumsko KOmpromisno Rangiranje (VIKOR) and DEMATEL. The relatively small number of works should also be pointed out in which the geometric model of aggregation was used, represented by the Weighted Product (WP) method and some MAUT/MAVT implementations. On the other hand, among the methods based on the outranking relation, the methods from the PROMETHEE and ELECTRE groups are primarily used. The NAIADE method and the methods of disaggregation of preferences based on rough sets, that is, RS/DRSA, are used much less frequently. The following methods are also rarely used: Goal Programming/Multi-Objective Decision-Making (GP/MODM) and DEA. When analysing the cited publications, one should also note the great importance of the methods implementing the fuzzy set theory, including: Fuzzy AHP (FAHP), Fuzzy ANP (FANP), and FTOPSIS, which are relatively often used to solve problems of sustainability. The analysis presented in Table 2 identified a set of discrete multi-criteria methods most commonly used in sustainability problems. Nevertheless, as mentioned earlier, the selection of a specific method should depend on its features and characteristics of the decision problem under consideration.

The analysis of multi-criteria methods in terms of the features characterized in Section 3 is presented in Table 3. Table 3 presents the methods most often used in sustainability problems, selected on the basis of the quantitative analysis presented in Table 2. In addition, Table 3 also characterizes the new methods used in the sustainability study, that is, PROmethee for Sustainability Assessment (PROSA-C) [67] and the New Easy Approach To Fuzzy PROMETHEE (NEAT F-PROMETHEE) [43].

The analysis presented in Table 3. shows that the use of multi-criteria methods in the assessment of sustainability gives much more flexibility than using one of the classic indices. The multi-criteria methods are also more functional than the classical sustainability indices in terms of almost all stages of the sustainability assessment process, discussed, for example, by Ibáñez-Forés et al. [10].

5. Discussion

The use of multi-criteria methods is associated with the need to select indicators and components of sustainability that act as criteria. On the one hand, it can be considered a disadvantage in relation to classic indices containing predefined sets of indicators. However, on the other hand, it allows you to choose criteria tailored to a specific decision problem, without excluding the possibility of using a predefined set of indicators, derived from one of the classic sustainability indices. It is important to be able to choose criteria adjusted to the scale of the problem, which is beneficial considering the fact that classic indices are most often intended for a global assessment of sustainability and sustainable development. Meanwhile, the use of multi-criteria methods and the selection of criteria related to it make it possible to prepare a decision model intended, for example, to evaluate individual activities or to develop a model covering more dimensions of sustainability.

The advantage of multi-criteria methods over classic indices is also manifested in the fact that these methods are characterized by a fully formalized calculation procedure, most often covering such aspects as: data normalization (most often it is a linear transformation or distance from the target), determination of criteria weights, and aggregation of performance alternatives based on different calculation models. Meanwhile, many indices skip the data normalization step, transferring this action to the index user. A series of indices also does not allow for the use of indicator weights, and the aggregation of assessments is carried out using only the arithmetic or geometric model.

In the context of aggregation models used in sustainability indices, it should be emphasized that arithmetic and geometric models are specific forms of additive and multiplicative value functions. Therefore, these models require at least the preferential independence of criteria or indicators. If, on the other hand, there are relationships between the indicators, then the results obtained with the use of these models may be erroneous [48]. Meanwhile, no classical sustainability index examines the relationship between sustainability indicators. It should be noted, however, that such relationships may often occur, if only due to the high complexity of decision problems related to sustainability [81,82]. In this respect, the advantage of some multi-criteria methods that do not require independence of criteria (methods using the outranking relation) or allow to take into account the dependencies between criteria in a decision problem (e.g., ANP) is visible.

Some multi-criteria methods also take into account uncertainty, in particular relating to: assessments (performance) of alternatives, criteria weights, and decision-maker preferences (e.g., the NEAT F-PROMETHEE method takes into account all three indicated types of uncertainty). Most often, multi-criteria methods also allow for the sensitivity analysis of the solution to changes in the weighting of criteria and the efficiency of alternatives [83]. The selected methods also offer great possibilities for the analysis and interpretation of the evaluation results. For example, together with the PROMETHEE methods, the Geometrical Analysis for Interactive Assistance (GAIA) method [70] is provided, which allows to consider a decision problem from the perspective of a description problematic (P.δ) [23].

6. Conclusions

Answering the research questions posed in the introduction, it should be pointed out that the most commonly used MCDA methods in sustainability problems are methods based on the additive model of criteria aggregation, such as AHP, WAM/SAW, TOPSIS, VIKOR. Methods that use an outranking relationship or a multiplicative aggregation model, such as PROMETHEE, ELECTRE, ANP or WP, are less commonly used. One can also observe the growing importance of methods implementing the theory of fuzzy sets. When it comes to the applicability of individual MCDA methods, it is obviously impossible to indicate any universal method that could be used in every decision-making problem. The methods should be applied according to the needs arising from the structure of the problem and the decision situation.

The contribution of the article is the MCDA method selection framework for the study of sustainability, sustainable development and sustainability assessment. The proposed framework allows the decision-maker to independently assess the potential of using individual multi-criteria methods in specific decision-making problems related to sustainability. The framework enables the selection of an appropriate MCDA method depending on the defined needs of the decision-maker, resulting from the decision problem, its structure and decision-making situation. The method chosen by the decision-maker can then be successfully applied to solve the decision problem. The method selected on the basis of reliable premises and adjusted to the defined needs will allow to obtain a reliable solution, satisfactory for the decision-maker.

As for research limitations, the basic limitation results from the huge number of MCDA methods developed and used in the literature. Contemporary publications mention over 200 MCDA methods. Therefore, it is obvious that the framework presented in this article does not contain a complete set of MCDA methods, but covers only the most popular and most commonly used methods in the literature. Therefore, further research should focus on supplementing the framework with other, less popular MCDA methods.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Tables

Table 1

Features of multi-criteria methods determining their applicability in problems of sustainability.

	Roy and Słowiński [16]	Wątróbski et al. [17]	Sadok et al. [18]	Rowley et al. [8]	De Montis et al. [19]	Moghaddam et al. [20]	Polatidis et al. [21]	Conelli et al. [3]
Problematic	x	x				x		x
Preference relations		x		x		x	x	x
Order of alternatives		x				x		x
Alternative preference scales	x	x	x		x	x	x	x
Type of criteria weights		x		x	x	x	x	x
Meaning of criteria weights				x		x		x
Compensation	x		x		x	x	x	x
Uncertainty	x	x			x		x	x
Relationship between criteria	x			x	x		x
Ease of the method’s use	x				x	x	x	x
Number of decision-makers				x	x	x	x
Software support						x		x

Table 2

Quantitative analysis of the applications of individual multi-criteria methods in the problems of sustainability.

Reference	[7]	[60]	[61]	[62]	[10]	[63]	[64]	[65]	[66]	Mean[%]
Class of Sustainability Problems	General [%]	General [%]	Infrastructure Investments [%]	Product Life Cycle [%]	Production Technologies [%]	Supply Chain [%]	Planning and Energy Management [%]
AHP	31.8	16.7	11.6	37.7	30.0	14.7	25.0	65.5	25.7	28.7
WAM/SAW	30.5	14.8	50.0	11.7	16.7	0.0	11.4	3.4	0.0	15.4
FS	0.0	9.3	9.3	0.0	20.0	47.1	13.6	3.4	12.2	12.8
PROMETHEE	3.1	5.6	2.3	10.4	13.3	0.0	13.6	0.0	12.2	6.7
TOPSIS	6.8	7.4	2.3	3.9	6.7	0.0	0.0	3.4	13.5	4.9
ELECTRE	4.5	3.7	0.0	2.6	10.0	0.0	6.8	10.3	4.1	4.7
ANP	4.5	7.4	5.8	2.6	3.3	17.6	0.0	0.0	0.0	4.6
FAHP	0.0	9.3	0.0	7.8	0.0	0.0	6.8	3.4	10.8	4.2
MAUT/MAVT	6.2	1.9	9.3	9.1	0.0	0.0	2.3	3.4	0.0	3.6
FTOPSIS	0.0	7.4	0.0	10.4	0.0	0.0	0.0	3.4	5.4	3.0
VIKOR	1.7	3.7	1.2	0.0	0.0	0.0	4.5	3.4	8.1	2.5
GP/MODM	3.8	3.7	3.5	0.0	0.0	8.8	0.0	0.0	0.0	2.2
DEA	2.4	0.0	0.0	0.0	0.0	5.9	4.5	0.0	0.0	1.4
NAIADE	2.1	1.9	0.0	2.6	0.0	0.0	4.5	0.0	0.0	1.2
RS/DRSA	1.4	1.9	0.0	0.0	0.0	2.9	0.0	0.0	0.0	0.7
DEMATEL	0.0	1.9	0.0	0.0	0.0	2.9	0.0	0.0	0.0	0.5
FANP	0.0	1.9	0.0	1.3	0.0	0.0	0.0	0.0	1.4	0.5
WP	1.4	1.9	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.4
Number of applications	292	54	86	77	30	68	44	29	74	Total 754

AHP—Analytic Hierarchy Process, WAM/SAW—Weighted Arithmetic Mean/Simple Additive Weighting, FS—Fuzzy Sets, PROMETHEE—Preference Ranking Organization METHod for Enrichment Evaluation, TOPSIS—Technique for Order of Preference by Similarity to Ideal Solution, ELECTRE—ELimination Et Choix Traduisant la REalité, ANP—Analytic Network Process, FAHP—Fuzzy AHP, MAUT/MAVT—Multiple-Attribute Utility Theory/Multiple-Attribute Value Theory, FTOPSIS—Fuzzy TOPSIS, VIKOR—VIšekriterijumsko KOmpromisno Rangiranje, GP/MODM—Goal Programming/Multi-Objective Decision-Making, DEA—Data Envelopment Analysis, NAIADE—Novel Approach to Imprecise Assessment and Decision Environments, RS/DRSA—Rough Sets/Dominance-based Rough Sets Approach, DEMATEL—Decision-Making Trial and Evaluation Laboratory, FANP—Fuzzy ANP, WP—Weighted Product.

Table 3

Analysis of the features of multi-criteria methods that determine their application in sustainability problems.

	Problematic	Preference Relations	Order of Alternatives	Scales of Alternative Performance	Type pf Criteria Weights	Significance of Weights	Compensation	Uncertainty	Relationships between Criteria	Ease of the Method Use	Number of Decision-Makers	Software Support	Operational Approach	Reference
AHP	γ	I,P	TO	QN	QN	TS	H	N	HD	M	MD	HH	SC	[68]
WAM/SAW	γ	I,P	TO	QN	QN	TS	F	N	N	E	PM	HH	SC	[69]
PROMETHEE I	γ	I,P,Q,R	PO	QN	QN	IC	L	UP	HD	E	MD	HH	OR	[70]
PROMETHEE II	γ	I,P,Q	TO	QN	QN	CT	M	UP	HD	E	MD	HH	OR	[70]
TOPSIS	γ	I,P	TO	QN	QN	TS	F	N	N	E	MD	LW	SC	[69]
ELECTRE I	α	(I,P)S,R	ST	QL	QN	IC	L	N	N	H	SD	LW	OR	[28]
ELECTRE IS	α	(I,P,Q)S,R	ST	QN	QN	IC	L	UP	N	H	SD	LW	OR	[28]
ELECTRE II	γ	(I,P)S,R	PO	QL	QN	IC	L	N	N	H	SD	LW	OR	[28]
ELECTRE III	γ	(I,P,Q)S,R	PO	QN	QN	IC	L	UP	N	H	SD	LW	OR	[28]
ELECTRE IV	γ	(I,P,Q)S,R	PO	QN	-	IC	L	UP	N	H	SD	LW	OR	[28]
ELECTRE TRI	β	(I,P,Q)S,R	CL	QN	QN	IC	L	UP	N	H	SD	LW	OR	[28]
ANP	γ	I,P	TO	QN	QN	IC	L	N	ID	M	MD	HH	SC	[71]
FAHP	γ	I,P	TO	QN	QN	TS	H	UE,UW	HD	M	MD	N	SC	[72]
MAUT	γ	I,P	TO	QN	QN	TS	F/L	UE	N	M	PM	LW	SC	[73]
MAVT	γ	I,P	TO	QN	QN	TS	F/L	N	N	E	PM	LW	SC	[73]
FTOPSIS	γ	I,P	TO	QN	QN	TS	F	UE,UW	N	M	MD	N	SC	[58]
VIKOR	γ	I,P	TO	QN	QN	TS	F	N	N	E	PM	N	SC	[74]
GP/MODM	ζ	-	-	QN	-	-	L	N	N	M	SD	N	-	[75]
DEA	γ	I,P	TO	QN	-	-	L	N	N	M	SD	LW	SC	[76]
NAIADE I	γ	S,R	PO	QN	-	-	L	UE	N	M	SD	LW	OR	[77]
NAIADE II	γ	S	TO	QN	-	-	M	UE	N	M	SD	LW	OR	[77]
RS/DRSA	γ	S	TO	QN	-	-	L	UP	HD	M	SD	LW	OR	[39]
DEMATEL	γ	I,P	TO	QL	QL	TS	H	N	ID	M	MD	N	SC	[78]
FANP	γ	I,P	TO	QN	QN	IC	L	UE,UW	ID	M	MD	N	SC	[79]
WP	γ	I,P	TO	QN	QN	IC	L	N	N	E	PM	N	SC	[80]
PROSA-C	γ	I,P,Q	TO	QN	QN	IC	L	UP	HD	M	PM	N	OR	[67]
NEAT F-PROMETHEE I	γ	I,P,Q,Q/P,J,R	PO	QN	QN	IC	L	UE,UW,UP	HD	M	PM	N	OR	[43]
NEAT F-PROMETHEE II	γ	I,P,Q,Q/P,J	TO	QN	QN	CT	M	UE,UW,UP	HD	M	PM	N	OR	[43]

Abbreviations: Problematic: α—selection, β—sorting, γ—ranking, ζ—design, Preference relations: I—indifference, P—strict preference, Q—weak preference, R—incomparability, S—outranking, Q/P—preference, J—J-preference, Alternative order: TO—full order, PO—partial order, ST—subset, CL—assignment to individual classes, Alternative performance scales: QL—qualitative, QN—quantitative, Criteria weight type:QL—qualitative, QN—quantitative, Weight significance: TS—trade-offs, IC—importance coefficients, CT—intermediate between trade-offs and importance coefficients, Compensation: F—full, H—high, M—mean/partial, L—low, Uncertainty: UE—uncertain evaluation/performance of alternatives, UW—uncertain weights, UP—uncertain preferences, N—none, Relationships between criteria: ID—network, HD—hierarchical, N—none, Ease of the method use: E—easy, M—medium, H—hard, Number of decision-makers: SD—single decision-maker, MD—many decision-makers, PM—possibility of easy extension to many decision-makers, Software support: HH—high, LW—low, N—none, Operational approach: SC—single synthesizing criterion, OR—outranking relation.

References

1. Singh, R.K.; Murty, H.R.; Gupta, S.K.; Dikshit, A.K. An Overview of Sustainability Assessment Methodologies. Ecol. Indic.; 2012; 15, pp. 281-299. [DOI: https://dx.doi.org/10.1016/j.ecolind.2011.01.007]

2. Jaehn, F. Sustainable Operations. Eur. J. Oper. Res.; 2016; 253, pp. 243-264. [DOI: https://dx.doi.org/10.1016/j.ejor.2016.02.046]

3. Cinelli, M.; Coles, S.R.; Kirwan, K. Analysis of the Potentials of Multi Criteria Decision Analysis Methods to Conduct Sustainability Assessment. Ecol. Indic.; 2014; 46, pp. 138-148. [DOI: https://dx.doi.org/10.1016/j.ecolind.2014.06.011]

4. White, L.; Lee, G.J. Operational Research and Sustainable Development: Tackling the Social Dimension. Eur. J. Oper. Res.; 2009; 193, pp. 683-692. [DOI: https://dx.doi.org/10.1016/j.ejor.2007.06.057]

5. Antunes, C.H.; Henriques, C.O. Multi-Objective Optimization and Multi-Criteria Analysis Models and Methods for Problems in the Energy Sector. Multiple Criteria Decision Analysis; International Series in Operations Research & Management Science Greco, S.; Ehrgott, M.; Figueira, J.R. Springer: New York, NY, USA, 2016; pp. 1067-1165. ISBN 978-1-4939-3093-7

6. Merad, M.; Dechy, N.; Serir, L.; Grabisch, M.; Marcel, F. Using a Multi-Criteria Decision Aid Methodology to Implement Sustainable Development Principles within an Organization. Eur. J. Oper. Res.; 2013; 224, pp. 603-613. [DOI: https://dx.doi.org/10.1016/j.ejor.2012.08.019]

7. Diaz-Balteiro, L.; González-Pachón, J.; Romero, C. Measuring Systems Sustainability with Multi-Criteria Methods: A Critical Review. Eur. J. Oper. Res.; 2017; 258, pp. 607-616. [DOI: https://dx.doi.org/10.1016/j.ejor.2016.08.075]

8. Rowley, H.V.; Peters, G.M.; Lundie, S.; Moore, S.J. Aggregating Sustainability Indicators: Beyond the Weighted Sum. J. Environ. Manag.; 2012; 111, pp. 24-33. [DOI: https://dx.doi.org/10.1016/j.jenvman.2012.05.004]

9. Tahseen, S.; Karney, B.W. Reviewing and Critiquing Published Approaches to the Sustainability Assessment of Hydropower. Renew. Sustain. Energy Rev.; 2017; 67, pp. 225-234. [DOI: https://dx.doi.org/10.1016/j.rser.2016.09.031]

10. Ibáñez-Forés, V.; Bovea, M.D.; Pérez-Belis, V. A Holistic Review of Applied Methodologies for Assessing and Selecting the Optimal Technological Alternative from a Sustainability Perspective. J. Clean. Prod.; 2014; 70, pp. 259-281. [DOI: https://dx.doi.org/10.1016/j.jclepro.2014.01.082]

11. Kumar, A.; Sah, B.; Singh, A.R.; Deng, Y.; He, X.; Kumar, P.; Bansal, R.C. A Review of Multi Criteria Decision Making (MCDM) towards Sustainable Renewable Energy Development. Renew. Sustain. Energy Rev.; 2017; 69, pp. 596-609. [DOI: https://dx.doi.org/10.1016/j.rser.2016.11.191]

12. Ziemba, P. Towards Strong Sustainability Management—A Generalized PROSA Method. Sustainability; 2019; 11, 1555. [DOI: https://dx.doi.org/10.3390/su11061555]

13. Mofidi Chelan, M.; Alijanpour, A.; Barani, H.; Motamedi, J.; Azadi, H.; Van Passel, S. Economic Sustainability Assessment in Semi-Steppe Rangelands. Sci. Total Environ.; 2018; 637–638, pp. 112-119. [DOI: https://dx.doi.org/10.1016/j.scitotenv.2018.04.428] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29751294]

14. Kornyshova, E.; Salinesi, C. MCDM Techniques Selection Approaches: State of the Art. Proceedings of the 2007 IEEE Symposium on Computational Intelligence in Multi-Criteria Decision-Making; Honolulu, HI, USA, 1–5 April 2007; pp. 22-29.

15. Ziemba, P. Selection of Electric Vehicles for the Needs of Sustainable Transport under Conditions of Uncertainty—A Comparative Study on Fuzzy MCDA Methods. Energies; 2021; 14, 7786. [DOI: https://dx.doi.org/10.3390/en14227786]

16. Roy, B.; Słowiński, R. Questions Guiding the Choice of a Multicriteria Decision Aiding Method. EURO J. Decis. Process.; 2013; 1, pp. 69-97. [DOI: https://dx.doi.org/10.1007/s40070-013-0004-7]

17. Wątróbski, J.; Jankowski, J.; Ziemba, P.; Karczmarczyk, A.; Zioło, M. Generalised Framework for Multi-Criteria Method Selection. Omega; 2018; 86, pp. 107-124. [DOI: https://dx.doi.org/10.1016/j.omega.2018.07.004]

18. Sadok, W.; Angevin, F.; Bergez, J.-É.; Bockstaller, C.; Colomb, B.; Guichard, L.; Reau, R.; Doré, T. Ex Ante Assessment of the Sustainability of Alternative Cropping Systems: Implications for Using Multi-Criteria Decision-Aid Methods. A Review. Agron. Sustain. Dev.; 2008; 28, pp. 163-174. [DOI: https://dx.doi.org/10.1051/agro:2007043]

19. De Montis, A.; De Toro, P.; Droste-Franke, B.; Omann, I.; Stagl, S. Assessing the Quality of Different MCDA Methods. Alternatives for Environmental Evaluation; Routledge: London, UK, 2005; pp. 99-133.

20. Moghaddam, N.B.; Nasiri, M.; Mousavi, S.M. An Appropriate Multiple Criteria Decision Making Method for Solving Electricity Planning Problems, Addressing Sustainability Issue. Int. J. Environ. Sci. Technol.; 2011; 8, pp. 605-620. [DOI: https://dx.doi.org/10.1007/BF03326246]

21. Polatidis, H.; Haralambopoulos, D.A.; Munda, G.; Vreeker, R. Selecting an Appropriate Multi-Criteria Decision Analysis Technique for Renewable Energy Planning. Energy Sources Part B Econ. Plan. Policy; 2006; 1, pp. 181-193. [DOI: https://dx.doi.org/10.1080/009083190881607]

22. Roy, B. Problematics as Guides in Decision Aiding. Multicriteria Methodology for Decision Aiding; Nonconvex Optimization and Its Applications Roy, B. Springer: Boston, MA, USA, 1996; pp. 57-74. ISBN 978-1-4757-2500-1

23. Ishizaka, A.; Nemery, P. General Introduction. Multi-Criteria Decision Analysis: Methods and Software; Wiley: Chichester, UK, 2013; pp. 1-9. ISBN 978-1-118-64489-8

24. Roy, B. Preference, Indifference, Incomparability: Binary Relations and Basic Structures. Multicriteria Methodology for Decision Aiding; Nonconvex Optimization and Its Applications Springer: Boston, MA, USA, 1996; pp. 79-126. ISBN 978-1-4419-4761-1

25. Ziemba, P. Multi-Criteria Approach to Stochastic and Fuzzy Uncertainty in the Selection of Electric Vehicles with High Social Acceptance. Expert Syst. Appl.; 2021; 173, 114686. [DOI: https://dx.doi.org/10.1016/j.eswa.2021.114686]

26. Bouyssou, D.; Vincke, P. Binary Relations and Preference Modeling. Decision-Making Process Concepts and Methods; Wiley-Blackwell: Hoboken, NJ, USA, 2010; pp. 49-84. ISBN 978-0-470-61187-6

27. Guitouni, A.; Martel, J.-M. Tentative Guidelines to Help Choosing an Appropriate MCDA Method. Eur. J. Oper. Res.; 1998; 109, pp. 501-521. [DOI: https://dx.doi.org/10.1016/S0377-2217(98)00073-3]

28. Figueira, J.R.; Mousseau, V.; Roy, B. Electre Methods. Multiple Criteria Decision Analysis; International Series in Operations Research & Management Science; Springer: New York, NY, USA, 2016; pp. 155-185. ISBN 978-1-4939-3093-7

29. Ziemba, P. Inter-Criteria Dependencies-Based Decision Support in the Sustainable Wind Energy Management. Energies; 2019; 12, 749. [DOI: https://dx.doi.org/10.3390/en12040749]

30. Roy, B. Paradigms and Challenges. Multiple Criteria Decision Analysis; International Series in Operations Research & Management Science; Greco, S.; Ehrgott, M.; Figueira, J.R. Springer: New York, NY, USA, 2016; pp. 19-39. ISBN 978-1-4939-3093-7

31. Whitaker, R. Criticisms of the Analytic Hierarchy Process: Why They Often Make No Sense. Math. Comput. Model.; 2007; 46, pp. 948-961. [DOI: https://dx.doi.org/10.1016/j.mcm.2007.03.016]

32. Guitouni, A.; Martel, J.-M.; Vincke, P. A Framework to Choose a Discrete Multicriterion Aggregation Procedure; Defence Research Establishment Valcatier (DREV): Québec, QC, Canada, 1998.

33. Munda, G. Multiple Criteria Decision Analysis and Sustainable Development. Multiple Criteria Decision Analysis: State of the Art Surveys; International Series in Operations Research & Management Science Greco, S.; Ehrgott, M.; Figueira, J.R. Springer: New York, NY, USA, 2016; pp. 1235-1267. ISBN 978-1-4939-3094-4

34. Roy, B. Coherent Criterion Family and Decision Aiding in the Description Problematic. Multicriteria Methodology for Decision Aiding; Nonconvex Optimization and Its Applications Roy, B. Springer: Boston, MA, USA, 1996; pp. 215-235. ISBN 978-1-4757-2500-1

35. Munda, G. “Measuring Sustainability”: A Multi-Criterion Framework. Environ. Dev. Sustain.; 2005; 7, pp. 117-134. [DOI: https://dx.doi.org/10.1007/s10668-003-4713-0]

36. Bouyssou, D. Some Remarks on the Notion of Compensation in MCDM. Eur. J. Oper. Res.; 1986; 26, pp. 150-160. [DOI: https://dx.doi.org/10.1016/0377-2217(86)90167-0]

37. Juwana, I.; Muttil, N.; Perera, B.J.C. Indicator-Based Water Sustainability Assessment—A Review. Sci. Total Environ.; 2012; 438, pp. 357-371. [DOI: https://dx.doi.org/10.1016/j.scitotenv.2012.08.093] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/23022721]

38. Bouyssou, D. Building Criteria: A Prerequisite for MCDA. Readings in Multiple Criteria Decision Aid; Bana e Costa, C.A. Springer: Berlin/Heidelberg, Germany, 1990; pp. 58-80.

39. Greco, S.; Matarazzo, B.; Słowiński, R. Decision Rule Approach. Multiple Criteria Decision Analysis: State of the Art Surveys; International Series in Operations Research & Management Science Greco, S.; Ehrgott, M.; Figueira, J.R. Springer: New York, NY, USA, 2016; pp. 497-552. ISBN 978-1-4939-3094-4

40. Ziemba, P. Uncertain Multi-Criteria Analysis of Offshore Wind Farms Projects Investments—Case Study of the Polish Economic Zone of the Baltic Sea. Appl. Energy; 2022; 309, 118232. [DOI: https://dx.doi.org/10.1016/j.apenergy.2021.118232]

41. Ziemba, P. Multi-Criteria Fuzzy Evaluation of the Planned Offshore Wind Farm Investments in Poland. Energies; 2021; 14, 978. [DOI: https://dx.doi.org/10.3390/en14040978]

42. Perny, P.; Roy, B. The Use of Fuzzy Outranking Relations in Preference Modelling. Fuzzy Sets Syst.; 1992; 49, pp. 33-53. [DOI: https://dx.doi.org/10.1016/0165-0114(92)90108-G]

43. Ziemba, P. Neat F-promethee—A New Fuzzy Multiple Criteria Decision Making Method Based on the Adjustment of Mapping Trapezoidal Fuzzy Numbers. Expert Syst. Appl.; 2018; 110, pp. 363-380. [DOI: https://dx.doi.org/10.1016/j.eswa.2018.06.008]

44. Carlsson, C.; Fullér, R. Multiple Criteria Decision Making: The Case for Interdependence. Comput. Oper. Res.; 1995; 22, pp. 251-260. [DOI: https://dx.doi.org/10.1016/0305-0548(94)E0023-Z]

45. Wu, Y.; Geng, S.; Xu, H.; Zhang, H. Study of Decision Framework of Wind Farm Project Plan Selection under Intuitionistic Fuzzy Set and Fuzzy Measure Environment. Energy Convers. Manag.; 2014; 87, pp. 274-284. [DOI: https://dx.doi.org/10.1016/j.enconman.2014.07.001]

46. Gölcük, İ.; Baykasoğlu, A. An Analysis of DEMATEL Approaches for Criteria Interaction Handling within ANP. Expert Syst. Appl.; 2016; 46, pp. 346-366. [DOI: https://dx.doi.org/10.1016/j.eswa.2015.10.041]

47. Lu, M.-T.; Lin, S.-W.; Tzeng, G.-H. Improving RFID Adoption in Taiwan’s Healthcare Industry Based on a DEMATEL Technique with a Hybrid MCDM Model. Decis. Support Syst.; 2013; 56, pp. 259-269. [DOI: https://dx.doi.org/10.1016/j.dss.2013.06.006]

48. Mandic, K.; Bobar, V.; Delibašić, B. Modeling Interactions Among Criteria in MCDM Methods: A Review. Decision Support Systems V—Big Data Analytics for Decision Making, Proceedings of the First International Conference, International Conference on Decision Support System Technology, Belgrade, Serbia, 27–29 May 2015; Delibašić, B.; Hernández, J.E.; Papathanasiou, J.; Dargam, F.; Zaraté, P.; Ribeiro, R.; Liu, S.; Linden, I. Springer: Berlin/Heidelberg, Germany, 2015; pp. 98-109.

49. Saaty, T.L. Fundamentals of the Analytic Network Process—Dependence and Feedback in Decision-Making with a Single Network. J. Syst. Sci. Syst. Eng.; 2004; 13, pp. 129-157. [DOI: https://dx.doi.org/10.1007/s11518-006-0158-y]

50. Al-Shemmeri, T.; Al-Kloub, B.; Pearman, A. Model Choice in Multicriteria Decision Aid. Eur. J. Oper. Res.; 1997; 97, pp. 550-560. [DOI: https://dx.doi.org/10.1016/S0377-2217(96)00277-9]

51. Løken, E. Use of Multicriteria Decision Analysis Methods for Energy Planning Problems. Renew. Sustain. Energy Rev.; 2007; 11, pp. 1584-1595. [DOI: https://dx.doi.org/10.1016/j.rser.2005.11.005]

52. Georgopoulou, E.; Lalas, D.; Papagiannakis, L. A Multicriteria Decision Aid Approach for Energy Planning Problems: The Case of Renewable Energy Option. Eur. J. Oper. Res.; 1997; 103, pp. 38-54. [DOI: https://dx.doi.org/10.1016/S0377-2217(96)00263-9]

53. Xu, G.; Liu, F. An Approach to Group Decision Making Based on Interval Multiplicative and Fuzzy Preference Relations by Using Projection. Appl. Math. Model.; 2013; 37, pp. 3929-3943. [DOI: https://dx.doi.org/10.1016/j.apm.2012.08.007]

54. Limayem, M.; Banerjee, P.; Ma, L. Impact of GDSS: Opening the Black Box. Decis. Support Syst.; 2006; 42, pp. 945-957. [DOI: https://dx.doi.org/10.1016/j.dss.2005.08.004]

55. Belton, V.; Pictet, J. A Framework for Group Decision Using a MCDA Model: Sharing, Aggregating or Comparing Individual Information?. J. Decis. Syst.; 1997; 6, pp. 283-303. [DOI: https://dx.doi.org/10.1080/12460125.1997.10511726]

56. Saaty, T.L. Decision Making—the Analytic Hierarchy and Network Processes (AHP/ANP). J. Syst. Sci. Syst. Eng.; 2004; 13, pp. 1-35. [DOI: https://dx.doi.org/10.1007/s11518-006-0151-5]

57. Shih, H.-S.; Shyur, H.-J.; Lee, E.S. An Extension of TOPSIS for Group Decision Making. Math. Comput. Model.; 2007; 45, pp. 801-813. [DOI: https://dx.doi.org/10.1016/j.mcm.2006.03.023]

58. Chen, C.-T.; Lin, C.-T.; Huang, S.-F. A Fuzzy Approach for Supplier Evaluation and Selection in Supply Chain Management. Int. J. Prod. Econ.; 2006; 102, pp. 289-301. [DOI: https://dx.doi.org/10.1016/j.ijpe.2005.03.009]

59. Li, Y.; Thomas, M.A. A Multiple Criteria Decision Analysis (MCDA) Software Selection Framework. Proceedings of the 2014 47th Hawaii International Conference on System Sciences; Washington, DC, USA, 6–9 January 2014; pp. 1084-1094.

60. Büyüközkan, G.; Karabulut, Y. Sustainability Performance Evaluation: Literature Review and Future Directions. J. Environ. Manag.; 2018; 217, pp. 253-267. [DOI: https://dx.doi.org/10.1016/j.jenvman.2018.03.064] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29605780]

61. Sierra, L.A.; Yepes, V.; Pellicer, E. A Review of Multi-Criteria Assessment of the Social Sustainability of Infrastructures. J. Clean. Prod.; 2018; 187, pp. 496-513. [DOI: https://dx.doi.org/10.1016/j.jclepro.2018.03.022]

62. Thies, C.; Kieckhäfer, K.; Spengler, T.S.; Sodhi, M.S. Operations Research for Sustainability Assessment of Products: A Review. Eur. J. Oper. Res.; 2018; 274, pp. 1-21. [DOI: https://dx.doi.org/10.1016/j.ejor.2018.04.039]

63. Rajeev, A.; Pati, R.K.; Padhi, S.S.; Govindan, K. Evolution of Sustainability in Supply Chain Management: A Literature Review. J. Clean. Prod.; 2017; 162, pp. 299-314. [DOI: https://dx.doi.org/10.1016/j.jclepro.2017.05.026]

64. Wimmler, C.; Hejazi, G.; Fernandes, E.d.O.; Moreira, C.; Connors, S. Multi-Criteria Decision Support Methods for Renewable Energy Systems on Islands. J. Clean Energy Technol.; 2015; 3, pp. 185-195. [DOI: https://dx.doi.org/10.7763/JOCET.2015.V3.193]

65. Vassoney, E.; Mammoliti Mochet, A.; Comoglio, C. Use of Multicriteria Analysis (MCA) for Sustainable Hydropower Planning and Management. J. Environ. Manag.; 2017; 196, pp. 48-55. [DOI: https://dx.doi.org/10.1016/j.jenvman.2017.02.067]

66. Siksnelyte, I.; Zavadskas, E.; Streimikiene, D.; Sharma, D.; Siksnelyte, I.; Zavadskas, E.K.; Streimikiene, D.; Sharma, D. An Overview of Multi-Criteria Decision-Making Methods in Dealing with Sustainable Energy Development Issues. Energies; 2018; 11, 2754. [DOI: https://dx.doi.org/10.3390/en11102754]

67. Ziemba, P. Multi-Criteria Stochastic Selection of Electric Vehicles for the Sustainable Development of Local Government and State Administration Units in Poland. Energies; 2020; 13, 6299. [DOI: https://dx.doi.org/10.3390/en13236299]

68. Saaty, T.L.; Vargas, L.G. Models, Methods, Concepts & Applications of the Analytic Hierarchy Process; 2nd ed. International Series in Operations Research & Management Science; Springer: New York, NY, USA, 2012; ISBN 978-1-4614-3596-9

69. Hwang, C.-L.; Yoon, K. Methods for Multiple Attribute Decision Making. Multiple Attribute Decision Making: Methods and Applications A State-of-the-Art Survey; Lecture Notes in Economics and Mathematical Systems Hwang, C.-L.; Yoon, K. Springer: Berlin/Heidelberg, Germany, 1981; pp. 58-191. ISBN 978-3-642-48318-9

70. Brans, J.-P.; De Smet, Y. Promethee Methods. Multiple Criteria Decision Analysis; International Series in Operations Research & Management Science Greco, S.; Ehrgott, M.; Figueira, J.R. Springer: New York, NY, USA, 2016; pp. 187-219. ISBN 978-1-4939-3093-7

71. Saaty, T.L.; Vargas, L.G. Decision Making with the Analytic Network Process: Economic, Political, Social and Technological Applications with Benefits, Opportunities, Costs and Risks; 2nd ed. International Series in Operations Research & Management Science; Springer: New York, NY, USA, 2013; ISBN 978-1-4614-7278-0

72. Mikhailov, L.; Tsvetinov, P. Evaluation of Services Using a Fuzzy Analytic Hierarchy Process. Appl. Soft Comput.; 2004; 5, pp. 23-33. [DOI: https://dx.doi.org/10.1016/j.asoc.2004.04.001]

73. Dyer, J.S. Multiattribute Utility Theory (MAUT). Multiple Criteria Decision Analysis; International Series in Operations Research & Management Science Greco, S.; Ehrgott, M.; Figueira, J.R. Springer: New York, NY, USA, 2016; pp. 285-314. ISBN 978-1-4939-3093-7

74. Opricovic, S.; Tzeng, G.-H. Extended VIKOR Method in Comparison with Outranking Methods. Eur. J. Oper. Res.; 2007; 178, pp. 514-529. [DOI: https://dx.doi.org/10.1016/j.ejor.2006.01.020]

75. Ishizaka, A.; Nemery, P. Goal Programming. Multi-Criteria Decision Analysis: Methods and Software; Wiley: Chichester, UK, 2013; pp. 222-234. ISBN 978-1-118-64489-8

76. Huguenin, J.-M. Data Envelopment Analysis. Multi-Criteria Decision Analysis: Methods and Software; Wiley: Chichester, UK, 2013; pp. 235-274. ISBN 978-1-118-64489-8

77. Munda, G. Multicriteria Evaluation in a Fuzzy Environment: The Naiade Method. Multicriteria Evaluation in a Fuzzy Environment: Theory and Applications in Ecological Economics; Contributions to Economics Physica-Verlag HD: Heidelberg, Germany, 1995; pp. 131-148. ISBN 978-3-642-49997-5

78. Hwang, C.-L.; Lin, M.-J. Experts Judgment and/or Group Participation. Group Decision Making under Multiple Criteria: Methods and Applications; Lecture Notes in Economics and Mathematical Systems Springer: Berlin/Heidelberg, Germany, 1987; pp. 145-305. ISBN 978-3-642-61580-1

79. Promentilla, M.A.B.; Furuichi, T.; Ishii, K.; Tanikawa, N. A Fuzzy Analytic Network Process for Multi-Criteria Evaluation of Contaminated Site Remedial Countermeasures. J. Environ. Manag.; 2008; 88, pp. 479-495. [DOI: https://dx.doi.org/10.1016/j.jenvman.2007.03.013]

80. Mateo, J.R.S.C. Weighted Sum Method and Weighted Product Method. Multi Criteria Analysis in the Renewable Energy Industry; Green Energy and Technology Springer: London, UK, 2012; pp. 19-22. ISBN 978-1-4471-2346-0

81. Wagenhals, S.; Garner, W.; Duckers, L.; Kuhn, K. Sustainability Index with Integrated Indicator Dependencies. Manag. Econ. Eng.; 2014; 12, pp. 15-29. [DOI: https://dx.doi.org/10.3846/bme.2014.02]

82. Tseng, M.-L.; Divinagracia, L.; Divinagracia, R. Evaluating Firm’s Sustainable Production Indicators in Uncertainty. Comput. Ind. Eng.; 2009; 57, pp. 1393-1403. [DOI: https://dx.doi.org/10.1016/j.cie.2009.07.009]

83. Triantaphyllou, E. A Sensitivity Analysis Approach for MCDM Methods. Multi-Criteria Decision Making Methods: A Comparative Study; Applied Optimization Springer: Boston, MA, USA, 2000; pp. 131-175. ISBN 978-1-4419-4838-0