Using Different Qualitative Scales in a Multi-Criteria Decision-Making Procedure

Abstract

Many decision problems manage linguistic information assessed through several ordered qualitative scales. In these contexts, the main problem arising is how to aggregate this qualitative information. In this paper, we present a multi-criteria decision-making procedure that ranks a set of alternatives assessed by means of a specific ordered qualitative scale for each criterion. These ordered qualitative scales can be non-uniform and be formed by a different number of linguistic terms. The proposed procedure follows an ordinal approach by means of the notion of ordinal proximity measure that assigns an ordinal degree of proximity to each pair of linguistic terms of the qualitative scales. To manage the ordinal degree of proximity from different ordered qualitative scales, we provide a homogenization process. We also introduce a stochastic approach to assess the robustness of the conclusions.

1. Introduction

Many disciplines such as Marketing, Tourism, Engineering, Health Economics or Medical Care address the problem of choosing the optimal choice from a set of alternatives which can be assessed regarding multiple criteria through several ordered qualitative scales (OQSs). Likewise, in the framework of decision-making problems, some situations require that experts express their assessments taking into account their knowledge or experience by means of specific OQSs formed by linguistic terms (see Delgado et al. [1], Herrera and Herrera-Viedma [2], Herrera et al. [3] and de Andrés et al. [4], among others).

The concept of linguistic term was introduced by Zadeh [5,6,7] when he defined the concept of linguistic variable as those variables whose values are words or terms from natural or artificial languages. For example, “ugly”, “fair” and “beautiful” could be linguistic terms for a linguistic variable Aesthetics, as these are not numerical values.

The use of linguistic assessments coming from of different OQSs is not infrequent. For instance, the Statistical office of the European Union (Eurostat) uses different OQSs in the EU Statistics on Income and Living Conditions (EU-SILC). In its questionnaires we can easily find OQSs formed by different linguistic terms. For instance, in the question Q19 (HS140), Eurostat uses the following 3-term OQS: {“a heavy burden”, “somewhat a burden”, “not burden at all”} for asking households about the financial burden of total housing cost and in the question Q26 (HS120), a 5-term OQS: {“with great difficulty”, “with difficulty”, “with some difficulty”, “fairly easily”, “easily”, “’very easily”} for analyzing the households’ difficulties to make ends meet. On the other hand, in clinical diagnosis is also needed to aggregate ordinal information from different OQSs. The Hospital Anxiety and Depression Scale (HADS) developed by Zigmond and Snaith [8] also uses different scales for measuring depression severity in primary care. Some of these scales are: {“very seldom”, “not often”, “sometimes”, “often”}, {“not at all”, “not often”, “usually", “definitely”}, or {“only occasionally”, “from time to time, but not too often”, “a lot of the time”, “a great deal of the time”}.

OQSs formed by linguistic terms are appropriate tools for collecting the agents’ opinions and judgments in situation of vagueness and imprecision. People are more comfortable using words rather than numbers to describe probabilities. In this sense, authors such as Zimmer [9,10] and Windschitl and Wells [11] point out that words are more natural than numbers, since verbal expressions of uncertainty are easily understood and, besides, they emerged long before the development of probability. On the other hand, agents’ opinions and judgments are generally imprecise, and therefore it would be misleading to represent them by precise numerical values (see Beyth-Marom [12], Wallsten et al. [13] and Teigen [14], among others).

A common practice when dealing with OQSs is to start by converting the qualitative information into numbers by assigning numerical values to each linguistic term of the scale. However, if done prematurely and arbitrarily, numerical codifications can produce negative effects. The first is concerned with the validity of translating an OQS into a numerical form. That practice introduces properties that were not present in the original linguistic scale and imposes more informative content on the qualitative scale. For example, when the numerical codifications are equidistant, it is possible to assume that there are the same differences between terms of scales. On the other hand, ordinal scales establish an ordered relationship between the objects being measured. In these scales, numerical codifications do not provide information on the magnitude of the differences between the terms; they only indicate different levels of the attribute or characteristics. Thus, in the ordinal scales, the numerical values obtained from measures of location and dispersion, such as arithmetic mean or variance are not meaningful, and they can lead to misinterpretation of results (see Merbitz et al. [15], Blair and Lacy [16], Franceschini [17], Bashkansky and Gadrich [18] and Gadrich et al. [19], among others).

In Franceschini et al. [20], it is shown that the arithmetic mean can distort the result obtained from ordinal data. To illustrate this issue, the authors consider the results of the visual control for a sample of 30 corks. The corks are assessed by the following 5-term OQS: {“reject”, “poor quality”, “medium quality”, “good quality”, “excellent quality”}. Initially, they assume that the scale is equally spaced and assign the numerical codification 1, 2, 3, 4, 5 to the corresponding linguistic terms. Considering this numerical codification, the arithmetic mean of the sample (3.7) indicates that the mean of the sample would be between “medium quality” and “good quality”. Afterwards, they consider different proximities between the terms of the scale and suggest a new numerical codification 1, 3, 9, 27, 81. In this case, the new arithmetic mean (32.9) provides a different result: the mean of the sample would be between “good quality” and “excellent quality”. Likewise, Stevens [21] pointed out that means and standard deviations ought not to be used with ordinal scales, since these statistics imply a knowing more than the relative rank-order data. In particular, the use of means and standard deviations computed on an ordinal scale is not reasonable when the consecutive intervals on the scale are unequal in size.

To overcome these issues, several methods have been proposed in the literature for managing linguistic information. For instance, those based on fuzzy techniques (see Herrera and Martínez [22], Herrera et al. [23], Wang [24], Liu and Jin [25] and Xiao et al. [26], among others), computing with words (see Zadeh [5,6,7,27], Li et al. [28], and Herrera et al. [29], among others) or dominance criteria and cumulative distribution functions (see Franceschini et al. [20,30] and Bashkansky-Gadrich [18]). However, the above procedures present some limitations. They do not consider how agents perceive the proximities between the linguistic terms of OQSs and, besides, some of them imply a direct conversion into cardinal values.

In general, OQSs are devised as balanced (they are formed by a fixed neutral central linguistic term and the rest of the terms are symmetrically distributed) and uniform (all the proximities between consecutive linguistic terms are considered identical). Nevertheless, in some cases the nature and the semantics of linguistic terms can be such that agents appreciate different proximities between the terms of the scale, i.e., they perceive the OQS as non-uniform. For instance, the OQS used in the HADS can be understood as non-uniform if agents perceive that “usually” is closer to “definitely” than to “not often”.

In the context of OQSs that are not uniformly and symmetrically distributed, Herrera et al. [23] developed a methodology to deal with unbalanced linguistic information. Their methodology is based on the concept of linguistic hierarchy and on the 2-tuple fuzzy linguistic representation model. This succeeds in allowing non-uniform scales in the 2-tuple fuzzy linguistic representation model, which is still equivalent to work with numerical values, as pointed out by García-Lapresta [31], requiring a qualitative to cardinal transformation at the outset.

In this paper, to deal with non-uniform OQSs in a way that postpones as much as possible the need to use numbers, we use ordinal proximity measures, introduced by García-Lapresta and Pérez-Román [32]. The concept of ordinal proximity measure takes into account psychological proximities between linguistic terms in a purely ordinal way by means of ordinal degrees. It is important to mention that these ordinal degrees are only abstract objects that represent different degrees of proximity. Likewise, we use the notion of metrizable ordinal proximity measure, which behaves as if the ordinal comparisons between the terms of the OQS were managed through a linear metric (see García-Lapresta et al. [33]). These concepts were recently implemented in some decision-making procedures where alternatives are always assessed through the same OQS. García-Lapresta and Pérez-Román [34] introduce a voting system that ranks the alternatives taking into account the medians of the ordinal degrees of proximity between the obtained individual assessments and the highest linguistic term of the scale. Later, García-Lapresta and González del Pozo [35] presented an ordinal multi-criteria decision-making (MCDM) procedure under uncertainty, extending the previous procedure to a multi-criteria setting, in which agents are allowed to assign two consecutive terms of the OQS to each alternative when they hesitate.

The aim of this paper is to propose a new MCDM procedure for managing ordinal information coming from several OQSs. This new procedure allows that each criterion uses a different OQS to evaluate the alternatives, thus making it more widely applicable. These OQSs can be considered to be non-uniform and can even be formed with a different number of linguistic terms for different criteria. The MCDM procedure yields a ranking of the alternatives based on an intuitive principle of compensation between advantages and disadvantages of each alternative in comparison with its opponents. To respect the ordinal information coming from the OQSs, the MCDM procedure follows an ordinal approach by means of the concept of ordinal proximity measure that avoid assigning arbitrary numerical codifications to the linguistic terms of the scales. In the proposed procedure, we also provide and analyze a homogenization process for managing the ordinal degrees of proximity from different OQSs.

The rest of this paper is organized as follows. Section 2 briefly introduces ordinal proximity measures. Section 3 presents a new MCDM procedure for ranking a set of alternatives assessed through a specific OQS for each criterion. Section 4 includes an example that illustrates how the proposed procedure works and we introduce a stochastic analysis to assess the robustness of the conclusions obtained. Likewise, we also include a comparison of our procedure with the methods SMAA-O (see Lahdelma [36]) and ZAPROS III (see Larichev [37]) that belongs to the family of verbal decision-making analysis. Finally, Section 5 presents some concluding remarks.

2. Preliminaries

Let us consider an OQSL={_l1,…,_lg}, withg≥3, arranged in ascending order,_l1≺⋯≺_lg.

We now recall the concept of ordinal proximity measure, introduced by García-Lapresta and Pérez-Román [32]. An ordinal proximity measure is a mapping that assigns an ordinal degree of proximity to each pair of linguistic terms of an OQSL. These ordinal degrees of proximity belong to a linear orderΔ={_δ1,…,_δh}, with_δ1≻⋯≻_δh, being_δ1and_δhthe maximum and the minimum degrees of proximity, respectively. The elements ofΔare not numbers and they only represent different degrees of proximity.

Definition 1

([32]). An ordinal proximity measure (OPM) onLwith values in Δ is a mappingπ:L×L⟶Δ, whereπ(_lr,_ls)=_πrsrepresents the degree of proximity between_lrand_ls, satisfying the following conditions:

1.

Exhaustiveness: For everyδ∈Δ, there exist_lr,_ls∈Lsuch thatδ=_πrs.

2.

Symmetry:_πsr=_πrs, for allr,s∈{1,…,g}.

3.

Maximum proximity:_πrs=_δ1⇔r=s, for allr,s∈{1,…,g}.

4.

Monotonicity:_πrs≻_πrtand_πst≻_πrt, for allr,s,t∈{1,…,g}such thatr<s<t.

Every OPM can be represented by ag×gsymmetric matrix with coefficients inΔ, where the elements in the main diagonal are_πrr=_δ1,r=1,…,g:

_π11⋯_π1s⋯_π1g⋯⋯⋯⋯⋯_πr1⋯_πrs⋯_πrg⋯⋯⋯⋯⋯_πg1⋯_πgs⋯_πgg.

This matrix is called proximity matrix associated withπ.

A prominent class of OPMs, introduced by García-Lapresta et al. [33], is the one of metrizable OPMs which is based on linear metrics on OQSs.

Definition 2

([33]). A linear metric onLis a mappingd:L×L⟶Rsatisfying the following conditions for allr,s,t∈{1,…,g}:

1.

Positiveness:d(_lr,_ls)≥0.

2.

Identity of indiscernibles:d(_lr,_ls)=0⇔_lr=_ls.

3.

Symmetry:d(_ls,_lr)=d(_lr,_ls).

4.

Linearity:d(_lr,_lt)=d(_lr,_ls)+d(_ls,_lt)wheneverr<s<t.

Definition 3

([33]). An OPMπ:L×L⟶Δis metrizable if there exists a linear metricd:L×L⟶Rsuch that_πrs≻_πtu⇔d(_lr,_ls)<d(_lt,_lu), for allr,s,t,u∈{1,…,g}.

Consequently, an OPM is metrizable if the ordinal comparisons between linguistic terms were made as if the agent had in mind a linear metric.

Forg=3there are three OPMs and all of them are metrizable. However, wheng>3 the complexity of constructing OPMs increases. In these cases, we can apply an algorithm that generates metrizable OPMs (see García-Lapresta et al. [33]). To do that, the algorithm is based on appropriate sequences of questions whose answers lead to metrizable OPMs. Forg=4there are 51 OPMs, but only 25 of them are metrizable.

3. The MCDM Procedure

In this section, we present an MCDM procedure for ranking a set of alternativesX={_x1,…,_xn}, withn≥2that are assessed by a group of agentsA={_a1,…,_am}, withm≥2, regarding a set of different criteriaC={_c1,…,_cq} . MCDM authors have proposed many methods for determining criteria weights (see, for instance, Solymosi and Dombi [38], Borcherding et al. [39], Diakoulaki et al. [40], Marichal and Roubens [41], Figueira and Roy [42], and Kao [43], among others). To focus on the main ideas in this contribution, we assume that the weights are exogenously given and they are collected in a weighting vector(_w1,…,_wq)∈^[0,1]q, with_w1+⋯+_wq=1. This is a common situation for evaluation panels and competition juries in which the weights are given in the regulations. Nevertheless, the procedure can easily be adapted to the case in which each agent uses different weights.

Agents assess the alternatives through a specific OQS for each criterion_ck∈C,^Lk={_l1k,…,_lgkk}, equipped with a metrizable OPM^πk:^Lk×^Lk⟶^Δk, where^Δk={_δ1k,…,_δhkk}. To focus on the main ideas in this contribution, we assume that all agents use the same scale and the same OPMs (one for each criterion). Again, this assumption can easily be dropped. Assuming the agents have the same OPMs means that they need to agree on the ordinal proximity levels before making their assessments of the alternatives, which is an essential step to ensure they attach the same meaning to the possible grades in the scale.

The proposed procedure is related to the Copeland rule (see Copeland [44]) and to the net flow method used in PROMETHEE II (see Brans [45], Brans and Vincke [46], Bouyssou [47] and Brans and De Smet [48]). The Copeland rule is a voting system that ranks order the alternatives taking into account the number of pairwise victories minus the number of pairwise defeats. The method PROMETHEE II generates a ranking considering positive and negative flows for each alternative according to the given criteria weights. The positive flow shows how much an alternative dominates over the others (the higher the positive flow, the better the alternative). The negative flow shows the dominance of all the other alternatives over the considered one (the smaller the negative flow, the better the alternative).

The assessments provided by the agents to the alternatives with respect to the criterion_ckare collected in a profile^Vk, a matrix of m rows and n columns of linguistic terms, where_via,k∈^Lkis the assessment given by the agent a to the alternative_xiregarding the criterion_ck:

^Vk=_v11,k⋯_vi1,k⋯_vn1,k⋯⋯⋯⋯⋯_v1a,k⋯_via,k⋯_vna,k⋯⋯⋯⋯⋯_v1m,k⋯_vim,k⋯_vnm,k.

3.1. The Procedure To rank the alternatives, the procedure is divided into the following steps:

• Gather the agents’ assessments in the corresponding profiles^V1,…,^Vq.

• Calculate, for every pair of alternatives_xi,_xj∈X, the ordinal degrees of proximity between the assessments given by the agents,^πk_via,k,_vja,k, for alla∈Aand_ck∈C.

• Homogenize the ordinal degrees of proximity coming from the metrizable OPMs considered in the different criteria by means of a mappingρ:_⋃k=1q ^Δk⟶[0,1]. Such mappingρmust satisfy the following conditions for everyk∈{1,…,q}:

  • Min-normalization:ρ_δ1k=0.

  • Max-normalization:ρ_δhkk=1.

  • Strict monotonicity:ρ_δr+1k>ρ_δrk,∀r∈{1,…,_hk−1}.

• Assign a score to the alternatives, through the mappingS:X⟶Rdefined as

  S(_xi)=∑a=1m∑k=1_via,k≻_vja,kq_wk·ρ^πk_via,k,_vja,k−∑a=1m∑k=1_via,k≺_vja,kq_wk·ρ^πk_via,k,_vja,k.

  Following the approaches of the Copeland rule and the PROMETHEE II,S(_xi) is divided into two parts. The first part of Equation (1) considers, for the criterion k and for the agent a, the victories of the linguistic assessment_via,kover the rest of assessments_vja,k(the higher, the better), and the second part the defeats of the linguistic assessment_via,kover the rest of assessments_vja,k(the lower, the better).

  In Equation (1),^πk_via,k,_vja,kmeasures for the criterion k and for the agent a, the proximity between the assessments_via,kand_vja,k. By means of these ordinal degrees of proximity,S(_xi)considers the scope of victories and defeats. The mappingρconverts the ordinal degrees of proximity to the interval [0,1]. Then, the final score is calculated multiplying the obtained results by the corresponding weights.

• Order the alternatives through the weak order ⪰ on X:

  _xi⪰_xj⇔S(_xi)≥S(_xj).

We now enumerate some properties that the proposed procedure satisfies:

• Anonymity: All agents are treated equally by the procedure.
• Neutrality: All alternatives are treated equally by the procedure.
• Monotonicity: If an agent improves the evaluation of an alternative on some criterion, all else remaining equal, then its score increases.

• Cancelation: In the uniform case, when two agents a and b increase and decrease at the same time their assessments, in such a way that agent a increases the assessment_via,k=_lrto_lr+1and agent b decreases the assessment_vib,k=_lrto_lr−1, for some alternative_xiand criterion_ck, thenS(_xi)does not change.

Definition 4.

Given a metrizable OPMπ:L×L⟶Δ, ρ preserves OPM linearity ifρ(_πrt)=ρ(_πrs)+ρ(_πst), for allr,s,t∈{1,…,g}such thatr<s<t.

Remark 1.

If ρ preserves OPM linearity, then one can interpret the proximities between levels as the difference of proximities between such levels and the worst performance as follows:

ρ^πk(_via,k,_vja,k)=ρ^πk(_l1k,_via,k)−ρ^πk(_l1k,_vja,k),if_via,k⪰_vja,k,ρ^πk(_l1k,_vja,k)−ρ^πk(_l1k,_via,k),if_via,k≺_vja,k.

Moreover, this allows rewriting Equation (1) as:

S(_xi)=∑a=1m∑k=1_via,k≻_vja,kq_wk·ρ^πk(_l1k,_via,k)−ρ^πk(_l1k,_vja,k)−∑a=1m∑k=1_via,k≺_vja,kq_wk·ρ^πk(_l1k,_vja,k)−ρ^πk(_l1k,_via,k)=∑a=1mn·∑k=1q_wk·ρ^πk(_l1k,_via,k)−∑k=1q∑j=1n_wk·ρ^πk(_l1k,_vja,k).

Since the rightmost summation is a constant that does not depend on_xi, ifρpreserves OPM linearity then its overall evaluation depends only on

∑a=1m∑k=1q_wk·ρ^πk(_l1k,_via,k).

Remark 2.

The simplest mapping considers that consecutive qualitative levels on a given criterion always represent the same difference, as follows:

ρ(_δrk)=r−1_hk−1.

This mapping preserves OPM linearity if the OPM is uniform, but not in the general case.

3.2. Linear Programming Formulation

A linear program can be solved to obtain aρmapping that preserves OPM linearity. The following formulation does so by maximizing the minimum difference between consecutive levels, i.e.,

maxminr=2,…,_hkρ(_δrk)−ρ(_δr−1k).

This can be formulated as a linear program by introducing an auxiliary variableσ.

To simplify notation, let_ρr,skdenoteρ(_πrsk). The main decision variables are theρvariables corresponding to the consecutive levels,_ρ1,2k,_ρ2,3k,…,_ρgk−1,gkk, plus the auxiliary variableσ.

Linear Program:

maxσsubject to:∑r=c−1c_ρr−1,rk=_ρc−2,ck, for c=3,…,_gk[C1]∑r=c−2c_ρr−1,rk=_ρc−3,ck, for c=4,…,_gk[C2]…∑r=2_gkρr−1,rk=_ρ1,gkk=1[C3]_ρr,sk=_ρt,uk,∀r,s,t,u:r>s∧t>u∧_πrsk=_πtuk[C4]_ρr,sk≥_ρt,uk+σ,∀r,s,t,u:r>s∧t>u∧_πrsk>_πtuk[C5]_ρr,sk≤_ρt,uk−σ,∀r,s,t,u:r>s∧t>u∧_πrsk<_πtuk[C6]_ρr−1,rk≥σ, for r=2,…,_gk[C7]

In this linear program, the first group of constraints, [C1], indicates that every subset of three consecutive levels_lc−2k,_lc−1k,_lckmust be such that_{ρc−2,c−1k}+_ρc−1,ck=_ρc−2,ck; a second group of constraints, [C2], indicates that every subset of four consecutive levels_lc−3k,_lc−2k,_lc−1k,_lckmust be such that_{ρc−3,c−2k}+_{ρc−2,c−1k}+_ρc−1,ck=_ρc−3,ck, and so on, until constraint [C3]. These constraints together ensure linearity. For instance, if a scale has 5 linguistic terms(_gk=5), then these linearity constraints would be:_ρ1,2k+_ρ2,3k=_ρ1,3k,_ρ2,3k+_ρ3,4k=_ρ2,4k,_ρ3,4k+_ρ4,5k=_ρ3,5k([C1]),_ρ1,2k+_ρ2,3k+_ρ3,4k=_ρ1,4k,_ρ2,3k+_ρ3,4k+_ρ4,5k=_ρ2,5k([C2]), and_ρ1,2k+_ρ2,3k+_ρ3,4k+_ρ4,5k=_ρ1,5k=1([C3]).

Then, constraints [C4-C6] are needed to ensure the monotonicity of theρmapping. Constraint [C4] can be used to reduce the number of variables in the linear program. Finally, constraint [C7] requires theρmapping to be positive.

Proposition 1.

If an OPM is uniform, then Equation (4) is an optimal solution for the above linear program.

Proof

Given several levels_gk, a uniform OPM requires_hk=_gk proximity degrees and Equation (4) yieldsρ(_δ1k)=0,ρ(_δ2k)=1_gk−1,ρ(_δ3k)=2_gk−1,…,ρ(_δgkk)=1. In such OPMs,_π12k=_π23k,_π23k=_π24k, etc., which by constraint [C4] implies_ρ1,2k=_ρ2,3k, etc. Then, constraint [C3] implies_{∑r=2gk ρr−1,rk}=(_gk−1)_ρ1,2k=1 , which means that the solution provided by Equation (4) is the only one satisfying these constraints.

Moreover,_{ρc−2,c−1k}=_ρc−1,ck=ρ(_δ2k)and_ρc−2,ck=ρ(_δ3k)=2ρ(_δ2k) , hence the constraints [C1] are satisfied. Similarly, all other linearity constraints [C2] until [C3] are naturally satisfied. Therefore, Equation (4) is a feasible solution to the linear program and, since it is the only feasible solution, it is also optimal. □

3.3. Stochastic Analysis

As mentioned before, theρ mapping can be obtained by a formula (e.g., Equation (4), which considers equal differences between any consecutive qualitative levels on a given criterion), or it can result from an optimization process enforcing linearity. However, in general there can be multiple otherρmappings preserving linearity. To analyze the results corresponding to the multipleρ mappings that preserve linearity it is possible to follow a stochastic approach, inspired by the Stochastic Multi-attribute Acceptability Analysis (SMAA) methods [36,49,50,51].

In the stochastic analysis, one samples randomly manyρmappings, and then results are computed for each of these mappings. This allows obtaining statistics about the results, such as:

• The rank acceptability indices_rip(for each alternative_xi, for each ranking position p), which is the probability (in terms or relative frequency) that alternative_xiis ranked in position p (i,p=1,…,n).

• The pairwise winning indicesp(i,j)(for each pair of alternatives(_xi,_xj)), which is the probability (in terms or relative frequency) that alternative_xiis ranked better than_xj(i,j=1,…,n).

To generate a randomρmapping preserving linearity it is sufficient to generate randomly theρvalues corresponding to the differences in the consecutive levels_ρ1,2k,_ρ2,3k,…,_ρgk−1,gkk. In one extreme case, when the consecutive levels are all different, the number of random numbers to generate is equal to_gk−1. In the other extreme case, when the consecutive levels are all equal, no random number needs to be generated. The needed random numbers can be sorted from lowest to highest and assigned in this order to the_πi,i+1k, from the lowest to the highest corresponding_πi,i+1k . When needed, a hit-and-run approach [50] can be followed to exclude cases in which theρvalues obtained by summing the obtained numbers do not respect the ordinal relation between the remaining_πrsk. For the cases in which the ordinal relations are respected, the generated random values can be divided by the sum_ρ1,2k+_ρ2,3k+⋯+_ρgk−1,gkkto normalize this sum to unity.

4. Practical Applications

To show how the proposed procedure works, we present an illustrative example that considers different proximities between the terms of OQSs. We also introduce a stochastic approach to assess the robustness of the conclusions obtained. Lastly, we apply our procedure to rank a selection of Research and Development (R&D) projects and compare its results with the ones obtained applying the methods SMAA-O (see Lahdelma [36]) and ZAPROS III (see Larichev [37]).

4.1. An Illustrative Example In this subsection, we apply the MCDM procedure to rank a set of alternatives. To do that, we consider three hotels which have been assessed by six agents taking into account three criteria: location, comfort of the rooms and service.

Table 1 shows the OQSs used for evaluating each criterion. To determine the criteria weights in hotel selection, we consider an empirical study where the most important criteria are those related to rooms such as: comfort and equipment of rooms, followed by the service and location (see Zaman et al. [52]). The different weights associated with each criterion are presented in Table 2 and the assessments given by the agents over the hotels are collected in Table 3.

To illustrate the proposed procedure, we consider three different cases. First, the uniform case; and then, two non-uniform cases.

4.1.1. The Uniform Case

The uniform case considers the metrizable OPMs^π1,^π2and^π3associated with the proximity matrices_A22,_A2222and_A222:

_A22=_{δ11δ21δ31δ11δ21δ11},_A2222=_{δ12δ22δ32δ42δ52δ12δ22δ32δ42δ12δ22δ32δ12δ22δ12},

_A222=_{δ13δ23δ33δ43δ13δ23δ33δ13δ23δ13}

that can be visualized in Figure 1. The subindices of the matrices^A′scorrespond to the subindices of the^δ′sappearing in the coefficients just over the main diagonal. We follow this pattern to denote subsequent matrices.

Table 4 shows for Hotel 1 theπ values obtained taking into account the above metrizable OPMs and the linguistic assessments contained in Table 3.

After applying the procedure to Hotel 1 produces the following results:

Location:

0.20·−12+12−12−12−12+0+0+12−22−12+0+0=−0.50.

Rooms:

0.50·−14+14+24+0−14−14+14+14+0−14+14+14=0.375.

Service:

0.30·13+13−13−13+13+23−13+13−23−13+13+0=0.10.

Thus, for Hotel 1 we obtainS(_H1)=−0.5+0.375+0.10=−0.025.

After applying the procedure to Hotels 2 and 3, we haveS(_H2)=−0.025andS(_H3)=0.05. Then, Hotel 3 ≻ Hotel 1 ∼ Hotel 2.

4.1.2. A Non-Uniform Case 1

In this case, we continue taking into account the same OPM,_A222 , for the third scale contained in Table 1. However, we consider two different ways to devise the other scales. These latter scales are equipped with the metrizable OPMs associated with the proximity matrices_A23,_A2332:

_A23=_{δ11δ21δ41δ11δ31δ11},_A2332=_{δ12δ22δ42δ62δ72δ12δ32δ52δ62δ12δ32δ42δ12δ22δ12},

_A222=_{δ13δ23δ33δ43δ13δ23δ33δ13δ23δ13}

that can be visualized in Figure 2.

After applying the procedure, we obtainS(_H1)=−0.017,S(_H2)=0.083andS(_H3)=−0.067. Then, Hotel 2 ≻ Hotel 1 ≻ Hotel 3.

4.1.3. A Non-Uniform Case 2

In this second case we consider that the three OQSs are not uniform. The scales are equipped with the metrizable OPMs associated with the following proximity matrices_A32,_A2332and_A233:

_A32=_{δ11δ31δ41δ11δ21δ11},_A2332=_{δ12δ22δ42δ62δ72δ12δ32δ52δ62δ12δ32δ42δ12δ22δ12},

_A233=_{δ13δ23δ43δ63δ13δ33δ53δ13δ33δ13}.

The OPMs considered in the non-uniform case 2 can be visualized in Figure 3.

When we apply the proposed procedure, we obtain the following results:S(_H1)=0.07,S(_H2)=−0.03andS(_H3)=−0.04. Then, Hotel 1 ≻ Hotel 2 ≻ Hotel 3.

Table 5 summarizes the results obtained after applying the procedure in the three cases considered. As can be seen, the rankings of the three hotels are different depending on how the proximities between the terms of the scales are perceived. This makes it extremely important that the agents debate which OPM best matches their interpretation of the linguistic terms, so that if proximities are not uniform then an appropriate non-uniform OPM is used instead.

4.1.4. Stochastic Analysis of the Non-Uniform Case 2

To illustrate the use of the stochastic analysis (Section 3.3), let us consider the non-uniform case 2 (Section 4.1.3). As depicted in Figure 3, for each scale only two random numbers need to be drawn: one to representρ(_δ2k)and another one to representρ(_δ3k)(values are drawn independently for each scale). These numbers are normalized dividing them byρ(_δ21)+ρ(_δ21)in the case of_A32, dividing them by2·ρ(_δ22)+2·ρ(_δ22)in the case of_A2332, and dividing them byρ(_δ23)+2·ρ(_δ23)in the case of_A233. All other ordinality constraints are respected (e.g.,ρ(_δ53)is surely greater thanρ(_δ43)becauseρ(_δ33)+ρ(_δ33)>ρ(_δ23)+ρ(_δ33)). The sampling was performed by running 10 000 iterations in a Monte-Carlo simulation.

Table 6 presents the rank acceptability indices for the three alternatives. Hotel 1 wins most of the times and is never ranked third (null acceptability for rank 3), whereas Hotel 2 never wins but nevertheless stays in the second position more often than in the third. Hotel 3 can be placed in any of the three positions, although it only reached the best position in less than 11% of the cases and its most likely position is the worst one.

These results can be further clarified by observing the pairwise winning indices on Table 7. It is possible to see why Hotel 2 cannot win: Hotel 1 beat it in all the simulations. The few cases in which Hotel 1 does not win correspond to the cases in which it is beaten by Hotel 3. This analysis indicates that in this example, the choice of Hotel 1, as indicated in the right column of Table 5, is fairly robust to changes in the definition of theρmapping. However, the conclusion that Hotel 2 is the second best one is less robust.

These results can be compared to the results that SMAA-O [36] would provide for the same data. SMAA-O is a stochastic approach for ordinal data that samples values compatible with an ordinal scale of g levels in a straightforward way: level_l1corresponds to value 0; level_lgcorresponds to value 1; levels_l2to_lg−1are assigned tog−2random values in ascending order. A uniform distribution in [0,1] is used for this purpose.

Applying SMAA-O to this case yields the results presented in Table 8 and Table 9.

Without the ability to exploit the knowledge of the ordinal proximities between the levels associated with each criterion, SMAA-O is not able to point any alternative as being the best one. Indeed, SMAA-O does not place any constraints on the mapping of qualitative scales other than respecting their order, which in this case as in many other ones is insufficient to make a clear distinction between the alternatives. Thus, when the decision makers define the criteria OPMs, the stochastic approach proposed in this paper is preferable. 4.2. Application of the Proposed MCDM Procedure to a Project Selection

In this subsection, we apply the proposed MCDM procedure to rank a selection of R&D projects and compare its results with the ones obtained applying the method ZAPROS III (see Larichev [37]). ZAPROS III belongs to the family of verbal decision analysis methods, which are applied to problems that use qualitative factors, especially difficult for formalization and numerical measurement.

Following the example presented by Larichev [37], we consider a decision maker that evaluates nine R&D projects regarding three different criteria. The OQSs used for evaluating the criteria are shown in Table 10. Although the three criteria have three levels, the OQSs are completely different and therefore it would be a strong assumption to use the same OPM for all these criteria. The assessments given by the decision maker to the projects are collected in Table 11.

The method ZAPROS III constructs a partial order over the set of the alternatives. ZAPROS III is based on elicitation of preferences around values that represent distances between the assessments on the ordinal scales of two criteria. However, this method does not use a specific weight for each criterion and does not consider the proximity between the linguistic terms of the scales.

To illustrate our procedure and compare its result with the output of ZAPROS III, we assume that all criteria have the same importance and consider in each criterion the following metrizable OPMs associated with the matrix_A23,_A32and_A32 , respectively (see Figure 4).

Figure 5 shows the partial order of the projects obtained through the method ZAPROS III, and in Table 12 the results obtained after applying the proposed MCDM procedure.

The results of the procedure presented in this paper are not directly comparable with ZAPROS as these methods follow a different logic. It is noteworthy that results are in agreement concerning which are the best alternatives (_x1and_x9) and also concerning which are the worst alternatives (_x5,_x6and_x8). Nevertheless, some differences can be highlighted. The first one is that our procedure always yields a weak order, whereas ZAPROS III finds some alternatives are not comparable (e.g.,_x2vs_x3), i.e., it can yield a partial order. Another important difference can be found in the comparison of the two top contenders,_x1and_x9. Without the ability to exploit information about ordinal proximities, ZAPROS III places_x9better than_x1. By exploiting the information contained in the OPMs, our approach places_x1better than_x9 . Given the OPMs in Figure 4 (for different OPMs the result changes), their proximity in Originality (advantage of_x1) is less (i.e., they are more distant) than the proximity in Prospects (the Qualification level being equal). Therefore, it is natural to conclude_x1is preferred to_x9.

5. Concluding Remarks The main contribution of this paper is to provide a MCDM procedure for ranking alternatives assessed by a group of agents regarding multiple criteria through several OQSs, which can use different terms and even have different number of levels. To illustrate how the MCDM procedure works, we have ranked a set of hotels considering different proximities between the terms of the OQSs. We have also introduced a stochastic approach to assess the robustness of the conclusions. It is important to mention that the MCDM procedure presents some advantages with respect to other proposals that deal with qualitative information:

• The proposed procedure is developed in a multi-criteria setting where alternatives are assessed by means of a specific OQS for each criterion. This is an important difference regarding other MCDM procedures that use the same qualitative scale in all criteria.
• The procedure preserves and respects ordinal information of the OQSs by means of the concept of ordinal proximity measure that takes into account how agents perceive the proximities between the linguistic terms of the scales.
• The possibility of applying the procedure in real decision-making problems such as: clinical diagnosis, quality control, customer satisfaction measurement, etc.

Regarding future research, it could be interesting to extend the proposed procedure to an uncertainty context, where agents are allowed to express their assessment by means of several consecutive linguistic terms instead of a single linguistic term (see García-Lapresta and González del Pozo [35]).

Location			Rooms			Service
l11	Poor		l12	Very uncomfortable		l13	Poor
l21	Acceptable		l22	Somewhat uncomfortable		l23	Fair
l31	Good		l32	Neither comfortable nor uncomfortable		l33	Good
			l42	Somewhat comfortable		l43	Excellent
			l52	Very comfortable

Weights
w1	0.20
w2	0.50
w3	0.30

	Hotel 1	Hotel 2	Hotel 3			Hotel 1	Hotel 2	Hotel 3
Location	l21	l31	l21		Location	l31	l21	l21
Rooms	l42	l52	l32		Rooms	l52	l42	l42
Service	l23	l13	l33		Service	l43	l33	l33
Assessments given by the agent 1.					Assessments given by the agent 2.
	Hotel 1	Hotel 2	Hotel 3			Hotel 1	Hotel 2	Hotel 3
Location	l11	l21	l31		Location	l21	l31	l31
Rooms	l52	l32	l52		Rooms	l32	l32	l42
Service	l13	l23	l33		Service	l23	l33	l33
Assessments given by the agent 3.					Assessments given by the agent 4.
	Hotel 1	Hotel 2	Hotel 3			Hotel 1	Hotel 2	Hotel 3
Location	l21	l31	l21		Location	l31	l31	l31
Rooms	l32	l42	l22		Rooms	l42	l52	l32
Service	l43	l33	l32		Service	l33	l13	l33
Assessments given by the agent 5.					Assessments given by the agent 6.

		Agent 1			Agent 2			Agent 3
		vi	vj	π(vi,vj)	vi	vj	π(vi,vj)	vi	vj	π(vi,vj)
H1 vs. H2	Location	l21	l31	δ2	l31	l21	δ2	l11	l21	δ2
	Rooms	l42	l52	δ2	l52	l42	δ2	l52	l32	δ3
	Service	l23	l13	δ2	l43	l33	δ2	l13	l23	δ2
H1 vs. H3	Location	l21	l21	δ1	l31	l21	δ2	l11	l31	δ3
	Rooms	l42	l32	δ2	l52	l42	δ2	l52	l52	δ1
	Service	l23	l33	δ2	l43	l33	δ2	l13	l33	δ3
		Agent 4			Agent 5			Agent 6
		vi	vj	π(vi,vj)	vi	vj	π(vi,vj)	vi	vj	π(vi,vj)
H1 vs. H2	Location	l21	l31	δ2	l21	l31	δ2	l31	l31	δ1
	Rooms	l32	l32	δ1	l32	l42	δ2	l42	l52	δ2
	Service	l23	l33	δ2	l43	l33	δ2	l33	l13	δ3
H1 vs. H3	Location	l21	l31	δ2	l21	l21	δ1	l31	l31	δ1
	Rooms	l32	l42	δ2	l32	l22	δ2	l42	l32	δ2
	Service	l23	l33	δ2	l43	l33	δ2	l33	l33	δ1

The Uniform Case	Non-Uniform Case 1	Non-Uniform Case 2
Hotel 3	Hotel 2	Hotel 1
Hotel 1 Hotel 2	Hotel 1	Hotel 2
	Hotel 3	Hotel 3

Alternative	Rank 1 (Best)	Rank 2	Rank 3
Hotel 1	0.892	0.108	0
Hotel 2	0	0.543	0.457
Hotel 3	0.108	0.349	0.543

	Hotel 1	Hotel 2	Hotel 3
Hotel 1	-	1.00	0.892
Hotel 2	0.00	-	0.543
Hotel 3	0.108	0.457	-

Alternative	Rank 1 (Best)	Rank 2	Rank 3
Hotel 1	0.340	0.486	0.174
Hotel 2	0.321	0.263	0.416
Hotel 3	0.339	0.251	0.410

	Hotel 1	Hotel 2	Hotel 3
Hotel 1	-	0.569	0.517
Hotel 2	0.431	-	0.501
Hotel 3	0.483	0.499	-

Originality			Prospects			Qualification of the Applicant
l11	Further development of previous ideas		l12	Success is hardly probable		l13	Unknown
l21	There are new elements in the proposal		l22	Success is rather probable		l23	Normal
l31	Absolutely new idea and/or approach		l32	High probability of success		l33	High

	Originality	Prospects	Qualification
x1	l31	l22	l23
x2	l21	l22	l33
x3	l11	l32	l23
x4	l31	l12	l33
x5	l21	l32	l13
x6	l11	l22	l23
x7	l11	l22	l33
x8	l11	l12	l33
x9	l21	l32	l23

Proposed Procedure
x1
x9
x2x4
x3x7
x5
x6
x8

Author Contributions

The authors contributed equally to this work. All authors have read and agreed to the published version of the manuscript.

Funding

Raquel González del Pozo and José Luis García-Lapresta acknowledge the financial support of the Spanish Ministerio de Economía y Competitividad (project ECO2016-77900-P) and ERDF. Raquel González del Pozo acknowledges the support of Movilidad Doctorandos UVa 2018. Luis Dias's work was funded by Fundação para a Ciência e a Tecnologia, Portugal (FCT) and the European Regional Development Fund (ERDF) under projects SET-LCA (CENTRO-01-0145-FEDER-030570) and UIDB/05037/2020.

Acknowledgments

The authors are grateful to two anonymous referees for their comments and suggestions.

Conflicts of Interest

The authors declare no conflict of interest.