Measuring Conflicts Using Cardinal Ranking: An Application to Decision Analytic Conflict Evaluations

Abstract

One of the core complexities involved in evaluating decision alternatives in the area of public decision-making is to deal with conflicts. The stakeholders affected by and involved in the decision often have conflicting preferences regarding the actions under consideration. For an executive authority, these differences of opinion can be problematic, during both implementation and communication, even though the decision is rational with respect to an attribute set perceived to represent social welfare. It is therefore important to involve the stakeholders in the process and to get an understanding of their preferences. Otherwise, the stakeholder disagreement can lead to costly conflicts. One way of approaching this problem is to provide means for comprehensive, yet effective stakeholder preference elicitation methods, where the stakeholders can state their preferences with respect to actions part of the current agenda of a government. In this paper we contribute two supporting methods: (i) an application of the cardinal ranking (CAR) method for preference elicitation for conflict evaluations and (ii) two conflict indices for measuring stakeholder conflicts. The application of the CAR method utilizes a do nothing alternative to differentiate between positive and negative actions. The elicited preferences can then be used as input to the two conflict indices indicating the level of conflict within a stakeholder group or between two stakeholder groups. The contributed methods are demonstrated in a real-life example carried out in the municipality of Upplands Väsby, Sweden. We show how a questionnaire can be used to elicit preferences with CAR and how the indices can be used to semantically describe the level of consensus and conflict regarding a certain attribute. As such, we show how the methods can provide decision aid in the clarification of controversies.

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1. Introduction

One of the core complexities involved in evaluating decision alternatives in the area of public decision-making is to deal with conflicts. The stakeholders affected by and involved in the decision often have conflicting preferences regarding the actions under consideration. For an executive authority, these differences of opinion can be problematic during both implementation and communication, even though the decision is rational with respect to an attribute set perceived to represent social welfare; see, e.g., [1–3].

Of particular interest for the decision-making forum are preferential conflicts between stakeholders, since such conflicts may cause delays in the decision process due to obstructions, hassles, and/or locked negotiations [4–6]. For instance, Hansson et al. [6] describe that the development plans of Husby, a suburb to Stockholm, had been on hold for several years due to conflicts. Another example is given in Danielson et al. [5], where three infrastructure decisions in Nacka, a municipality in the Stockholm region, were delayed for several years due to conflicts between stakeholders. Therefore, to avoid stakeholder conflicts it is important for the executive authority to involve the stakeholders in the process and to get an understanding of their preferences [7, 8].

This calls for a desire to become better informed with regard to potential controversies, and it has been discussed in contemporary decision analysis literature that interaction with stakeholders using web-based techniques is one feasible approach to obtain stakeholder preferences and then inform decision-makers by utilizing decision analysis techniques. For instance, French et al. [1] suggest that web-based approaches can be used to support and structure the democratic process. Hansson et al. [6] suggest that a solution to the above-mentioned problem of conflicts could be to actively interact with stakeholders early in the process by the use of social media.

In this approach, an important first step is to elicit the opinions or attitudes of the citizens regarding a set of “actions”. These are potential actions that in the future may be redefined into more well-defined projects. It has been recognized that the selected methods must be easy to use for less experienced stakeholders and at the same time be sufficiently powerful to enable them to provide meaningful feedback [9]. To facilitate scalable elicitation, many approaches (including ours) promote the use of web-based surveys or questionnaires distributed to stakeholders early in the planning phase [3].

In this paper we contribute with an application of the cardinal ranking (CAR) method [10], extended with a feature for conflict evaluations, and a method for measuring conflict within a stakeholder group and between two stakeholder groups. The contributions support the decision process in a public decision-making setting.

Cardinal Ranking for Conflict Evaluations. To the best of our knowledge, no method exists that utilizes preferences elicited by a cardinal ranking approach in conflict evaluations. In this application of CAR [10], the respondents, in addition to cardinally ranking the elements, state the performance of the elements relative to a do nothing alternative. For example, supporting statements are like “action 1 is better than action 2, and both actions are negative relative the do nothing alternative and “action 2 is much better than action 3, action 2 is positive whereas Action 3 is negative relative the do nothing alternative”.

Conflict Indices. In a group decision analysis setting, it is of interest to investigate whether a certain alternative is conflict-prone with respect to one or more attributes. The preferences elicited by the application of CAR can be used for assessing the level of conflict within or between stakeholder groups. We describe two such indices, the Within-Group Conflict Index and the Between-Group Conflict Index, both utilizing a sum of squares, used in Ward’s clustering method [11, p. 466] and extended with a positive stakeholder scaling constant. The underlying group formation is similar to the collective attractiveness and unattractiveness indices presented by [12]. The approaches differ in how the value functions are created and utilized. Bana e Costa [12] creates one value function per criterion by applying MACBETH and its semantic categories. In our approach the respondents use CAR to create their individual value functions per criterion. In turn, this enables individual value estimates of the do nothing alternative which is not necessarily viewed as a “neutral” alternative.

With this we contribute to the first important step in a participatory group decision process by elucidating both conflict-prone actions and nonconflicting actions. An action is conflict-prone when there are strong opposing preferences regarding the performance of the action, either within a stakeholder group or between two stakeholder groups. An action is a nonconflicting action when the stakeholders have similar preferences.

1.1. Case Setting

In this paper, we describe a case conducted in Upplands Väsby municipality slightly north of Stockholm City. In this case, we analyzed the citizens preferences regarding a set of actions that could be implemented in the future. A previous reporting of the case can be found in Chapter 7 of Ekenberg et al. [13], in which a simplified conflict measure was utilized. In the reported method, the stakeholders were divided into two groups, the con- and the pro-group. The conflict index was then measured as the difference between the arithmetic means of the groups part-worth values. The methods presented in this paper are rather based on Ward’s method [11].

Typically, the actions are described quite briefly, for example, in short sentences such as “Build residential area near the lake” or “Build apartments in the town centre”. In the paper, we use the term “action” to distinguish from the more traditional term “alternatives” employed within the field of decision analysis. In our setting, an action is a tentative project proposal without an associated cost but rather a line of direction for a future project. This is somewhat similar to the tentative proposals defined as “topics” by [14].

In order to understand attitudes, the questionnaire must allow for the participating stakeholders to state positive or negative preferences regarding an action’s performance relative to a do nothing action. In order to measure conflict, the statements need to be represented in a manner allowing for such a measure to be meaningful. Thus, the attitudes must be represented using a measurable value function where the questionnaire is used as a tool for eliciting preferences. Conventional preference elicitation methods for approximating such value functions are typically considered to be cognitively demanding [15]. The nature of the actions being so loosely defined renders the use of more elaborate preference elicitation techniques, since they rely on well-defined alternatives. To resolve this situation, the questionnaire we propose is inspired by the attitude surveys often employing different versions of the Likert scale.

An alternative approach to reducing the cognitive burden of the decision-maker is to use preference disaggregation techniques. The techniques do this by utilizing global preferences, e.g., a ranking of a subset of alternatives, or by a pairwise comparison of alternatives, to infer value functions; see, e.g., [16–19] for details.

The questionnaire, which was previously reported in [13, Ch. 7], enables the capturing of negative and positive attitudes of the actions relative to a do nothing action. Then, methods for cardinal ranking are used to interpret the responses in terms of surrogate values and attribute weights. Lastly, the questionnaire contains a section where the respondent enters demographic information. This information can then be used to analyze the preferences of the citizens, e.g., to find stakeholder groups with conflicting interests.

2. Modeling of Preferences

In the decision analysis field, we distinguish between two types of preference representation functions, the value function and the utility function. A value function represents preferences over certain outcomes, as opposed to the utility function which represents preferences over lotteries with uncertain outcomes, and the two representations are based upon different axiom systems. The focus in this paper is on preference representations under certainty, and in the absence of uncertain consequences (or risk), preferences over objects $x$ are typically represented by means of a value function $v(x)\mapsto [\mathrm{0,1}]$, such that the object $x$ is preferred to object $y$ (commonly denoted by $x\succ y$) by the decision-maker if and only if $v(x)>v(y)$. If $v(x)=v(y)$ then the decision-maker is indifferent between the two objects. The value function is measurable and is defined over an interval scale if $v(w)-v(x)>v(y)-v(z)$ and $v(w)>v(x),v(y)>v(z)$ entails that exchanging $x$ for $w$ is preferred compared to exchanging $z$ for $y$; see [20] for a comprehensive treatment.

In the case of multiple decision-makers or stakeholders $S_{\mathrm{1}},S_{\mathrm{2}},\dots ,S_n$ having different individual measurable value functions $v^{\mathrm{1}}(),v^{\mathrm{2}}(),\dots ,v^n()$, in group decision analysis, it is of concern to aggregate the set of individual value functions using some proper preference aggregation rule providing a group value function sharing the properties of the individual value functions. For this purpose, given that all individual value functions of stakeholder $k$ are of the form $v^k$ and that domain and range are shared among the stakeholders, it has been shown that an additive aggregation of individual measurable value functions provides a measurable group value function $W(A_j)$ for an alternative $A_j$, such that $W(A_j)=\sum {\lambda }_k{v}^k(A_j)$, where $v^k(A_j)$ is the value of alternative $A_j$ for stakeholder $S_k$, and ${\lambda }_k$ are stakeholder scaling constants subject to ${\lambda }_i\ge \mathrm{0}$ and $\sum {\lambda }_i=\mathrm{1}$ whenever exchange independence is satisfied (if all stakeholders are indifferent between two exchanges then the whole group must also be indifferent) [21].

In the multiattribute setting, considering a set of evaluation attributes $\mathit{G}=\{G_{\mathrm{1}},G_{\mathrm{2}},\dots ,G_m\}$, the value of an alternative $A_j$ is obtained by aggregating the corresponding set of value functions $v_{\mathrm{1}}(x_{\mathrm{1}}),v_{\mathrm{2}}(x_{\mathrm{2}}),\dots ,v_m(x_m)$. The most common way of aggregation is the additive approach such that the value $V(A_j)$ of $A_j$ is given by $V(A_j)={\sum }_i{w}_i{v}_{ij}$, where $v_{ij}$ is the value of alternative $A_j$ under attribute $G_i$, and $w_i$ is the weight of attribute $G_i$ under the condition that $\mathrm{0}\le w_i\le \mathrm{1}$ and ${\sum }_i{w}_i=\mathrm{1}$. This additive aggregation provides a measurable multiattribute value function given that the conditions of mutually preferential independence and difference independence hold; see, e.g., [22]. The weights $w_i$ then act as attribute scaling constants, scaling the value contribution to an alternative from an attribute. The scaled value $q_{ij}=w_i·v_{ij}$ is called the “part-worth” value of attribute $G_i$ to alternative $A_j$. In a group setting, the group’s value of alternative $A_j$ is then given by$$\begin{array}{}\text{(1)}& W\left(A_j\right)={\displaystyle \sum }_k{\lambda }_k{\displaystyle \sum }_i{q}_{ij}^k\end{array}$$

where $q_{ij}^k$ is the part-worth value of attribute $G_i$ to alternative $A_j$ for stakeholder $S_k$; i.e., $q_{ij}^k=w_i^k·v_{ij}^k$. Henceforth in this paper, $q_{ij}^k$ notation for a stakeholders part-worth value will be used, and we will denote the stakeholder $S_k$’s value of alternative $A_j$ under attribute $G_i$ with $v_{ij}^k$. In order to obtain these values, different approaches have been suggested to elicit them from decision-makers. For an overview of elicitation methods, see, e.g., [22, 23].

3. Rank-Based Elicitation

The procedure in rank-ordering methods is to elicit the preferences as ranks, which are then converted into cardinal surrogate weights. Two such methods, rank-sum (RS) and rank-reciprocal (RR), are described in [24], and a third method the Rank Order Centroid (ROC) is described in [25, 26].

Rank-ordering methods have desirable advantages over more precise elicitation techniques, such as (i) being easier to elicit the vague preferences (less cognitively demanding) and (ii) having increased likelihood for a group to come to agreement [27, 28]. But rank ordered weight elicitation may be problematic. For example, Jia et al. [29] point out that, in a real-life setting, uncertainty may exist regarding both the magnitudes and ordering of weights, and even though information regarding the difference in importance may exist, the information is not considered in the transformation from rank orders into weights.

A method taking the difference in importance into consideration is the cardinal ranking (CAR) method [10, 30]. In CAR, a prerequisite is an ordinal ranking, to which cardinal information is added; see Figure 1 for a visualization of the difference between ordinal and cardinal ranking. The cardinal information is used to denote the strength of preference between pairs of elements in the ranking. This strength of preference is interpreted as the number of steps between each pair of elements on an underlying importance scale. The notation ${\succ }_i$ is used for describing this, where $i$ is the number of steps. The cardinality can be described by semantic expressions, e.g., obtained from a linguistic analysis,

 

${~}_0$ equally important, $\mathrm{0}$ steps

 

${\succ }_1$ slightly more important, $\mathrm{1}$ step

 

${\succ }_2$ more important, $\mathrm{2}$ steps

 

${\succ }_3$ much more important, $\mathrm{3}$ steps

[figure omitted; refer to PDF]

This enables the decision-maker to make statements such as the following:

(i)

Attribute A is equally important (${~}_{\mathrm{0}}$) as attribute B

(ii)

Attribute B is slightly more important (${\succ }_{\mathrm{1}}$) than attribute C

(iii)

Attribute C is more important (${\succ }_{\mathrm{2}}$) than attribute D

(iv)

Attribute D is much more important (${\succ }_{\mathrm{3}}$) than attribute E

Note that similar linguistic translations are commonly used in other MCDA methods such as AHP [31] and MACBETH [32, 33].

The CAR method has been demonstrated using both linear inequalities to represent cardinal ranking statements [34] and closed formulas for obtaining surrogate weights [10, 30]. More recently, rank-based methods have been suggested for probability elicitation as well, with a particular aim for use in time scarce environments [35].

4. Cardinal Ranking for Conflict Evaluations

In this section we introduce an application of CAR which captures negative or positive preferences with regard to an alternative’s performance relative to a do nothing alternative over a set of attributes. The method enables respondents to express the strength of preference between ordered pairs of alternatives using steps of preference intensities and at the same time express whether the alternatives are negative or positive relative to a do nothing alternative.

Assume that we have a set of alternatives $\mathit{A}=\{A_{\mathrm{1}},A_{\mathrm{2}},\dots ,A_n\}$ which are evaluated against a set of attributes $\mathit{G}=\{G_{\mathrm{1}},G_{\mathrm{2}},\dots ,G_m\}$. To capture the negative or positive preferences we need to introduce a do nothing alternative $A_{\alpha }$ to the set of alternatives. The do nothing alternative represents the current state; i.e., it should be considered whether the actions under consideration are better/worse than this alternative. See, e.g., Lahdelma et al. [36] for arguments supporting this technique.

We conform to the CAR method’s procedure for eliciting the alternatives’ values [10, 30] but extend it with a third step, the step where the do nothing alternative is inserted in the ranking:

(1)

An ordinal number is assigned to each position on the underlying measurement scale.

(2)

The underlying scale consists of $Q$ positions in decreasing order of importance. Alternative $A_i$ has a position $p(i)$ on the scale, such that $\mathrm{1}\le p(i)\le Q$, where $Q\ge m$. The strength, or cardinality, of preference $s_i$ between two adjacent alternatives $A_i{\succ }_{s_i}{A}_j$, is then $s_i=\left|p\left(i\right)-p\left(j\right)\right|$.

(3)

A do nothing alternative $A_{\alpha }$ is inserted into the ranking by providing it with a position $p$.

(4)

The cardinal ranking is normalized to a proportional $[\mathrm{0,1}]$-value scale according to the following equation: $$\begin{array}{}\text{(2)}& v_i^{\mathrm{C}\mathrm{A}\mathrm{R}}=\frac{Q-p\left(i\right)}{Q-\mathrm{1}}\end{array}$$

4.1. Example of CAR for Conflict Evaluations

Assume that a decision-maker evaluates the performance of four alternatives, $\{A_{\mathrm{1}},\dots ,A_{\mathrm{4}}\}$, with regard to attribute $C_{\mathrm{1}}$.

(1)

The decision-maker orders the alternatives as $A_{\mathrm{1}}\succ A_{\mathrm{2}}\succ A_{\mathrm{3}}\succ A_{\mathrm{4}}$.

(2)

He/she adds cardinal information to the ordinal ranking by introducing cardinality (strength of preference) steps between pairs of alternatives:

(i)

$A_{\mathrm{1}}$ is much better (${\succ }_{\mathrm{3}}$) than $A_{\mathrm{2}}$,

(ii)

$A_{\mathrm{2}}$ is better (${\succ }_{\mathrm{2}}$) than $A_{\mathrm{3}}$

(iii)

$A_{\mathrm{3}}$ is much better (${\succ }_{\mathrm{3}}$) than $A_{\mathrm{4}}$,

 

which gives the following cardinal rank $A_{\mathrm{1}}{\succ }_{\mathrm{3}}{A}_{\mathrm{2}}{\succ }_{\mathrm{2}}{A}_{\mathrm{3}}{\succ }_{\mathrm{3}}{A}_{\mathrm{4}}$; see Figure 2.

(3)

He/she states that alternatives $A_{\mathrm{1}}$ and $A_{\mathrm{2}}$ are considered to be positive and $A_{\mathrm{3}}$ and $A_{\mathrm{4}}$ negative relative to the do nothing alternative. The do nothing alternative $A_{\alpha }$ is therefore inserted at position $\mathrm{0}$, between $A_{\mathrm{2}}$ and $A_{\mathrm{3}}$, giving the following cardinal ranking, $A_{\mathrm{1}}{\succ }_{\mathrm{3}}{A}_{\mathrm{2}}{\succ }_{\mathrm{1}}{A}_{\alpha }{\succ }_{\mathrm{1}}{A}_{\mathrm{3}}{\succ }_{\mathrm{3}}{A}_{\mathrm{4}}$; see Figure 2.

(4)

The cardinal ranking results in the following positions on the underlying scale:

(i)

$A_{\mathrm{1}}$ at position $p(\mathrm{4})$,

(ii)

$A_{\mathrm{2}}$ at position $p(\mathrm{1})$,

(iii)

$A_{\alpha }$ at position $p(\mathrm{0})$,

(iv)

$A_{\mathrm{3}}$ at position $p(-\mathrm{1})$,

(v)

$A_{\mathrm{4}}$ at position $p(-\mathrm{4})$.

 

These positions are then mapped onto a proportional $[\mathrm{0,1}]$-value scale, giving the alternatives the following values, where $v_{ij}$ denotes alternative $A_j$:s value under attribute $G_i$. As [37] points out, the do nothing alternative is not always assigned a score of 0. Note that in our approach the do nothing alternative can be assigned a value between 0 and 1 depending on its position in the ordinal ranking. In this example, $Q=\mathrm{9}$ and $p(\mathrm{4})=\mathrm{1}$, $p(\mathrm{1})=\mathrm{4}$, $p(\mathrm{0})=\mathrm{5}$, $p(-\mathrm{1})=\mathrm{6}$ and $p(-\mathrm{4})=\mathrm{9}$.

(i)

$v_{\mathrm{4}}=\mathrm{1.000}$,

(ii)

$v_{\mathrm{3}}=\mathrm{0.625}$,

(iii)

$v_{\alpha }=\mathrm{0.500}$,

(iv)

$v_{\mathrm{2}}=\mathrm{0.375}$,

(v)

$v_{\mathrm{1}}=\mathrm{0.000}$.

[figure omitted; refer to PDF]

4.2. Conflicts

In a group decision analysis setting, it may be of interest to investigate whether a certain alternative is conflict-prone with respect to one or more attributes. The preferences elicited in the application of CAR for conflict evaluations can be used for assessing the level of conflict within or between stakeholder groups. In the following section, we will propose two such indices, the within-group conflict index and the between-group conflict index. Based upon these indices we can define consensus properties that textually describe the level of conflict associated with an attribute with respect to a specific alternative. A similar approach was presented by Bana e Costa [12].

Before presenting the indices, we stipulate the following conditions for the concept of conflict.

Definition 1 (conflict).

Given a set of stakeholders $\mathit{S}$ and a set of alternatives $\mathit{A}$, conflict exists in $\mathit{S}$ if there are two or more stakeholders in $\mathit{S}$ with positive scaling constants and these have differing preferences towards at least one alternative $A_i\in \mathit{A}$. Formally, there must exist $S_k,S_l\in \mathit{S}$ with ${\lambda }_k,{\lambda }_l>\mathrm{0}$ such that $v_j^k<v_{\alpha }^k$ and $v_j^l\ge v_{\alpha }^l$.

Definition 1 explicitly says that if all stakeholders share preferences towards an alternative, there is no conflict. In this sense it can be argued that this definition is a very strong definition of conflict. In the case of multiple attributes, Definition 1 can be extended to attribute conflict. Needless to say, given an attribute $G_i$ with a positive weight $w_i>\mathrm{0}$ we have that $v_{ij}^k<v_{i\alpha }^k$ implies $q_{ij}^k<q_{i\alpha }^k$ and $v_{ij}^k\ge v_{i\alpha }^k$ implies $q_{ij}^k\ge q_{i\alpha }^k$ since $q_{ij}^k=w_i{v}_{ij}^k$. We can now define the meaning behind attribute conflict and measurable conflict.

Definition 2 (attribute conflict).

Given a set of stakeholders $\mathit{S}$, a set of alternatives $\mathit{A}$, and a set of evaluation attributes $\mathit{G}$, conflict exists for $G_i\in \mathit{G}$ if there exist $S_k,S_l\in \mathit{S}$ with ${\lambda }_k,{\lambda }_l>\mathrm{0}$ such that $q_{ij}^k<q_{i\alpha }^k$ and $q_{ij}^l\ge q_{i\alpha }^l$.

4.2.1. Measurable Conflict

The definitions above do not consider different opinions on how “good” (or “bad”) two alternatives are given that they are both considered as productive (or counter-productive) to count for conflict. In other words, the definitions do not account for value differences, which lead us to measurable conflict.

In a two-stakeholder setting, given that attribute conflict exists for attribute $G_i$ such that $q_{ij}^k<q_{i\alpha }^k$ and $q_{ij}^l\ge q_{i\alpha }^l$, we will argue that it is reasonable to base a measure of conflict upon the value differences:$$\begin{array}{}\text{(3)}& d_{ij}^k=\left|q_{i\alpha }^k-q_{ij}^k\right|\end{array}$$

The intuition behind (3) is that stakeholder $S_k$ considers $A_j$ to be $\left|q_{i\alpha }^k-q_{ij}^k\right|$ worse than the do nothing alternative while another stakeholder $S_l$ considers $A_j$ to be $\left|q_{i\alpha }^l-q_{ij}^l\right|$ better than the do nothing alternative. Given two stakeholders $S_k,S_l$ and two alternatives $A_{\mathrm{1}},A_{\mathrm{2}}$ such that$$\begin{array}{}\text{(4)}& q_{i\mathrm{1}}^k\le q_{i\mathrm{2}}^k\le q_{i\alpha }^k\text{(5)}& q_{i\mathrm{1}}^l\ge q_{i\mathrm{2}}^l\ge q_{i\alpha }^l,\end{array}$$

then conflict cannot be smaller for $A_{\mathrm{2}}$ compared to $A_{\mathrm{1}}$. Further, conflict is larger for $A_{\mathrm{1}}$ compared to $A_{\mathrm{2}}$ if $S_k$ would rather exchange $A_{\mathrm{1}}$ for $A_{\alpha }$ than exchange $A_{\mathrm{2}}$ for $A_{\alpha }$, or if $S_l$ would rather exchange $A_{\alpha }$ for $A_{\mathrm{1}}$ than exchange $A_{\alpha }$ for $A_{\mathrm{2}}$.

However, a generalized measure of conflict should consider more than two stakeholders. In addition to measuring how far from the do nothing alternative two stakeholders in conflict are, we need to consider disparities between sets of stakeholders. For this reason we adopt a cluster distance approach to conflict measurement by partitioning the stakeholders into two sets based upon their preferences towards the alternative.

Let $\mathit{S}=\{S_{\mathrm{1}},S_{\mathrm{2}},\dots ,S_n\}$ be a set of stakeholders. For each attribute $G_i$ and alternative $A_j$, we partition the stakeholder set $\mathit{S}$ into two partitions, the con-group ${\mathit{S}}_{ij}^{-}$ and the pro-group ${\mathit{S}}_{ij}^{+}$. The members of the ${\mathit{S}}_{ij}^{-}$ evaluated $v_{ij}^k$ to be less than the value of the do nothing alternative $v_{i\alpha }^k$, and the stakeholders in ${\mathit{S}}_{ij}^{+}$ evaluated $v_{ij}^k$ to be greater than or equal to $v_{i\alpha }^k$; see the following equations:$$\begin{array}{}\text{(6)}& {\mathit{S}}_{ij}^{-}={\left\{S_k\in \mathit{S}:v_{ij}^k<v_{i\alpha }^k\right\}}_{k=\mathrm{1}}^n\text{(7)}& {\mathit{S}}_{ij}^{+}={\left\{S_k\in \mathit{S}:v_{ij}^k\ge v_{i\alpha }^k\right\}}_{k=\mathrm{1}}^n\end{array}$$

Hence, the do nothing alternative $A_{\alpha }$ is used to separate the stakeholders into either ${\mathit{S}}_{ij}^{-}$ or ${\mathit{S}}_{ij}^{+}$. The intuition behind the conflict index is to measure the distance between these two groups such that if two stakeholders $S_k\in {\mathit{S}}_{ij}^{-},S_l\in {\mathit{S}}_{ij}^{+}$ in general disagree to a large extent, i.e., they both have a large differences $\left|q_{i\alpha }^k-q_{ij}^k\right|$ and $\left|q_{i\alpha }^l-q_{ij}^l\right|$, then the conflict is greater than if they have smaller differences. Further, it can be argued that if the power balance between ${\mathit{S}}_{ij}^{-}$ and ${\mathit{S}}_{ij}^{+}$ is more equal, then the conflict is stronger since it has been demonstrated that less powerful stakeholders are more willing to accept an alternative even though they deem it counterproductive [38]. In our case, this is represented as follows: if the difference $$\begin{array}{}\text{(8)}& {\displaystyle \sum }_{S_k\in {\mathit{S}}_{ij}^{-}}{\lambda }_k-{\displaystyle \sum }_{S_l\in {\mathit{S}}_{ij}^{+}}{\lambda }_l\end{array}$$ is small, then the power balance is more equal entailing larger conflict.

To measure these properties reflecting the distance between the groups, we utilize three sums of squares similarly to how Ward’s clustering method measures distances between clusters [11, p. 466]. We obtain the sum of squared differences between each $d_{ij}^k$ and the group’s mean distance, i.e., the differences $$\begin{array}{}\text{(9)}& d_{ij}^k-\frac{{\sum }_{S_k\in {\mathit{S}}_{ij}}{d}_{ij}^k}{\left|{\mathit{S}}_{ij}\right|}\end{array}$$ for the con-group (12), the pro-group (13), and the combined group (14), respectively.

Definition 3 (within-group conflict index).

A within-group conflict index $d_{ij}^{\mathit{S}}$ for stakeholder set $\mathit{S}$ with two or more stakeholders, under attribute $G_i$ and alternative $A_j$, is given by$$\begin{array}{}\text{(10)}& d_{ij}^{\mathit{S}}=\sqrt{\beta \left(T_{ij}^{\mathit{S}}-\left(C_{ij}^{\mathit{S}}+P_{ij}^{\mathit{S}}\right)\right)}\end{array}$$where $$\begin{array}{}\text{(11)}& \beta =\frac{\mathrm{1}}{{\sum }_{S_k\in S}{{\lambda }_k}^{\mathrm{2}}}\text{(12)}& C_{ij}^{\mathit{S}}={\displaystyle \sum }_{S_k\in {\mathit{S}}_{ij}^{-}}{{\lambda }_k}^{\mathrm{2}}{\left(d_{ij}^k-\frac{{\sum }_{S_k\in {\mathit{S}}_{ij}^{-}}{d}_{ij}^k}{\left|{\mathit{S}}_{ij}^{-}\right|}\right)}^{\mathrm{2}}\text{(13)}& P_{ij}^{\mathit{S}}={\displaystyle \sum }_{S_k\in {\mathit{S}}_{ij}^{+}}{{\lambda }_k}^{\mathrm{2}}{\left(d_{ij}^k-\frac{{\sum }_{S_k\in {\mathit{S}}_{ij}^{+}}{d}_{ij}^k}{\left|{\mathit{S}}_{ij}^{+}\right|}\right)}^{\mathrm{2}}\text{(14)}& T_{ij}^{\mathit{S}}={\displaystyle \sum }_{S_k\in {\mathit{S}}_{ij}}{{\lambda }_k}^{\mathrm{2}}{\left(d_{ij}^k-\frac{{\sum }_{S_k\in {\mathit{S}}_{ij}}{d}_{ij}^k}{\left|{\mathit{S}}_{ij}\right|}\right)}^{\mathrm{2}}\end{array}$$

In (10), $\beta$ is used to normalize the value onto $[\mathrm{0,1}]$. It can be noted that $\mathrm{0}\le d_{ij}^{\mathit{S}}\le \mathrm{1}$, where $d_{ij}^{\mathit{S}}=\mathrm{1}$ occurs when all stakeholders in the con-group rank such that $q_{i\alpha }=\mathrm{1}$ and $q_{ij}=\mathrm{0}$, and all stakeholders in the pro-group rank such that $q_{i\alpha }=\mathrm{0}$ and $q_{ij}=\mathrm{1}$ and the power balance is equal. Further, $d_{ij}^{\mathit{S}}=\mathrm{0}$ occurs if either the con-group or the pro-group is empty.

4.3. Consensus Properties

The semantic meaning of the conflict indices can then be explained by dividing the conflict range $[\mathrm{0,1}]$ into a number of subintervals each describing a certain level of consensus. We define the points $a$ and $b$ as the interval’s lower and upper bound and stipulate $n$ interval points $\{x_{\mathrm{1}},x_{\mathrm{2}},\dots ,x_{n-\mathrm{1}},x_n\}$. These points are to divide the scale into the $n$ intervals such as $\left[a,x_{\mathrm{1}}\right],[x_{\mathrm{1}},x_{\mathrm{2}}],\dots ,[x_{n-\mathrm{1}},b]$. Each interval is then associated with a consensus property label describing the level of consensus, e.g., consensus aligned attribute, potentially controversial attribute, and controversial attribute, with regard to a specific action.

4.4. Example

Assume that ten stakeholders $\mathit{S}=\{S_{\mathrm{1}},\dots ,S_{\mathrm{10}}\}$ evaluate the performance of five actions $\{A_{\mathrm{1}},\dots ,A_{\mathrm{5}}\}$, with regard to attribute $G_{\mathrm{1}}$ using CAR for conflict evaluations. The actions’ positions on the underlying measurement are presented in Table 1 and the normalized values in Table 2. Attribute $G_{\mathrm{1}}$ is for illustrative purposes given the weight of 1.

Table 1

The positions on the underlying measurement scale produced by the cardinal rankings.

Stakeholder	Position
$$S_k$$	$$p(A_{\mathrm{1}})$$	$$p(A_{\mathrm{2}})$$	$$p(A_{\mathrm{3}})$$	$$p(A_{\mathrm{4}})$$	$$p(A_{\mathrm{5}})$$	$$p(A_{\alpha })$$
$${S_{}}_{\mathrm{1}}$$	8	6	8	1	1	8
$${S_{}}_{\mathrm{2}}$$	8	6	8	1	1	8
$${S_{}}_{\mathrm{3}}$$	8	7	8	2	1	8
$${S_{}}_{\mathrm{4}}$$	8	7	8	3	1	8
$${S_{}}_{\mathrm{5}}$$	8	7	8	3	1	8
$${S_{}}_{\mathrm{6}}$$	1	2	1	5	8	1
$${S_{}}_{\mathrm{7}}$$	1	2	1	5	8	1
$${S_{}}_{\mathrm{8}}$$	1	2	1	6	8	1
$${S_{}}_{\mathrm{9}}$$	1	3	1	7	8	1
$${S_{}}_{\mathrm{10}}$$	1	3	2	7	8	1

Table 2

The stakeholders’ cardinal rankings normalized to a proportional $[0,1]$-value scale.

Stakeholder	Value						Weight
$$S_k$$	$$v_{\mathrm{1,1}}^k$$	$$v_{\mathrm{1,2}}^k$$	$$v_{\mathrm{1,3}}^k$$	$$v_{\mathrm{1,4}}^k$$	$$v_{\mathrm{1,5}}^k$$	$$v_{1,\alpha }^k$$	$$w_1$$
$${S_{}}_{\mathrm{1}}$$	0.00	0.29	0.00	1.00	1.00	0.00	1.00
$${S_{}}_{\mathrm{2}}$$	0.00	0.29	0.00	1.00	1.00	0.00	1.00
$${S_{}}_{\mathrm{3}}$$	0.00	0.14	0.00	0.86	1.00	0.00	1.00
$${S_{}}_{\mathrm{4}}$$	0.00	0.14	0.00	0.71	1.00	0.00	1.00
$${S_{}}_{\mathrm{5}}$$	0.00	0.14	0.00	0.71	1.00	0.00	1.00
$${S_{}}_{\mathrm{6}}$$	1.00	0.86	1.00	0.43	0.00	1.00	1.00
$${S_{}}_{\mathrm{7}}$$	1.00	0.86	1.00	0.43	0.00	1.00	1.00
$${S_{}}_{\mathrm{8}}$$	1.00	0.86	1.00	0.29	0.00	1.00	1.00
$${S_{}}_{\mathrm{9}}$$	1.00	0.71	1.00	0.14	0.00	1.00	1.00
$${S_{}}_{\mathrm{10}}$$	1.00	0.71	0.86	0.14	0.00	1.00	1.00

Properties of $G_{\mathrm{1}}$ for each action can now be defined. First, we stipulate classes of controversy based upon the magnitude of conflict. Assume four such classes which range from “consensus aligned”, i.e., low conflict, to “very controversial” where the conflict is high; see Table 3. Note that the subranges and the semantical labels are defined by the authors for illustrative purposes. In a real setting the ranges and labels should be defined by the decision-maker.

Table 3

The table shows four conflict index intervals, each associated with a consensus property label which semantically describes the level of consensus.

Conflict index interval	Consensus property label
$$[0.00,0.10)$$	Consensus aligned
$$[0.10,0.20)$$	Potentially controversial
$$[0.20,0.50)$$	Controversial
$$[0.50,1.00]$$	Very controversial

To analyze conflict within the stakeholder group, for each action we form a con-group (6) and a pro-group (7) and subsequentially calculate the squared distance within the con-group (12) and the pro-group (13), the squared distance between the con- and pro-group (14), and the within-group conflict index $d_{ij}^{\mathit{S}}$ (10).

The results are shown in Table 4. Action $A_{\mathrm{1}}$ has $d_{\mathrm{1,1}}^{\mathit{S}}=\mathrm{0}$; i.e., there exists no conflict between the stakeholders since all stakeholders are members of the pro-group and no stakeholders are members of the con-group. As seen in Table 2, all stakeholders placed $A_{\mathrm{1}}$ on the same position as $A_{\alpha }$ in the cardinal ranking. However, stakeholders $S_{\mathrm{1}},\dots ,S_{\mathrm{5}}$ stated that all other actions were better than $A_{\mathrm{1}}$, and stakeholders $S_{\mathrm{6}},\dots ,S_{\mathrm{10}}$ stated that all other actions were worse than action $A_{\mathrm{1}}$. Thus, when the ranking was normalized to proportional $[\mathrm{0,1}]$-value scale, the value of action $A_{\mathrm{1}}$ for stakeholders $S_{\mathrm{1}},\dots ,S_{\mathrm{5}}$ was $v_{\mathrm{1,1}}^k$=0, and for stakeholders $S_{\mathrm{6}},\dots ,S_{\mathrm{10}}$ the value was $v_{\mathrm{1,1}}^k$=1. Action $A_{\mathrm{2}}$ has ${\mathrm{d}}_{\mathrm{1,2}}^{\mathit{S}}=\mathrm{0.2}$, since both the con-group and the pro-group estimated the value of action to lie close to $A_{\alpha }$. Action $A_{\mathrm{3}}$ has an even lower conflict index, $d_{\mathrm{1,3}}^{\mathit{S}}=\mathrm{0.0429}$, since all stakeholders except one are members of the pro-group. The pro-group members estimated the value of $A_{\mathrm{3}}$ to be equal to the value of $A_{\alpha }$, and the member of the con-group estimated the value of $A_{\mathrm{3}}$ to be slightly less than the value of $A_{\alpha }$. Action $A_{\mathrm{4}}$ has $d_{\mathrm{1,4}}^{\mathit{S}}=\mathrm{0.7857}$ since, as seen in Table 2, the stakeholders of both groups estimated it to lie closer towards the most preferred or least preferred alternative for both groups, action $A_{\mathrm{5}}$. Lastly, action $A_{\mathrm{5}}$ has $d_{\mathrm{1,5}}^{\mathit{S}}=\mathrm{1}$, since all members of the con- and pro-group estimate it to be the worst and, respectively, best action. See the classification of the actions in Table 5.

Table 4

The distance within the con- and pro-group, the distance between the con- and pro-group, and the conflict index.

	$$A_1$$	$$A_2$$	$$A_3$$	$$A_4$$	$$A_5$$
$$C_{1j}^{\mathit{S}}$$	0.0000	0.0002	0.0000	0.0008	0.0000
$$P_{1j}^{\mathit{S}}$$	0.0000	0.0002	0.0000	0.0008	0.0000
$$T_{1j}^{\mathit{S}}$$	0.0000	0.0045	0.0002	0.0634	0.1000
$${\mathit{d}}_{\mathit{1}\mathit{j}}^{\mathit{S}}$$	0.0000	0.2000	0.0429	0.7857	1.0000

Table 5

The classification of the actions consensus properties.

Consensus Property	$$A_{\mathrm{1}}$$	$$A_{\mathrm{2}}$$	$$A_{\mathrm{3}}$$	$$A_{\mathrm{4}}$$	$$A_{\mathrm{5}}$$
Consensus aligned	$$\times$$		$$\times$$
Potentially controversial		$$\times$$
Controversial
Very controversial				$$\times$$	$$\times$$

Note that the conflict index measures the disagreement between the con-group and the pro-group. If either the con-group or the pro-group is empty, $C_{ji}^S$ (or $P_{ij}^S$) cancels $T_{ij}^S$ out.

4.5. Between-Group Conflict Index

Of interest for the application case was first to measure the conflict within one group of stakeholders, i.e., the within-group conflict index. Second, it was of interest to measure the conflict between two stakeholder groups, e.g., people living in different parts of the town or belonging to different generations.

The between-group conflict index measures the conflict between two stakeholder groups. The intuition behind the between-group conflict index is that if the two groups disagree to the same extent on the same matters, there is low conflict between the groups. The between-group conflict should thus reveal to what extent two groups disagree differently from each other. Let $D$ and $E$ be two subsets of $\mathit{S}$. For each alternative $A_j$ and attribute $G_i$, we partition $D$ and $E$ into two partitions: the con-group ${\mathit{S}}_{ij}^{D-}$ and the pro-group ${\mathit{S}}_{ij}^{D+}$ for $D$, and the con-group ${\mathit{S}}_{ij}^{E-}$ and the pro-group ${\mathit{S}}_{ij}^{E+}$ for $E$. The members of ${\mathit{S}}_{ij}^{D-}$ and ${\mathit{S}}_{ij}^{E-}$ estimated the part-worth value $q_{ij}^k$ to be less than the part-worth value of the do nothing alternative $q_{i\alpha }^k$, and the members of ${\mathit{S}}_{ij}^{D+}$ and ${\mathit{S}}_{ij}^{E+}$ estimated it to be greater than or equal to $q_{i\alpha }^j$; see the following equation:$$\begin{array}{c}\text{(15)}& {\mathit{S}}_{ij}^{D-}={\left\{S_k\in D:q_{ij}^k<q_{i\alpha }^k\right\}}_{j=\mathrm{1}}^n{\mathit{S}}_{ij}^{D+}={\left\{S_k\in D:q_{ij}^k\ge q_{i\alpha }^k\right\}}_{j=\mathrm{1}}^n\\ {\mathit{S}}_{ij}^{E-}={\left\{S_k\in E:q_{ij}^k<q_{i\alpha }^k\right\}}_{j=\mathrm{1}}^n\\ {\mathit{S}}_{ij}^{E+}={\left\{S_k\in E:q_{ij}^k\ge q_{i\alpha }^k\right\}}_{j=\mathrm{1}}^n\end{array}$$

The squared distances within the con-group $C_{ij}^D,C_{ij}^E,C_{ij}^{D,E}$ (12), and the pro-group $P_{ij}^D,P_{ij}^E,P_{ij}^{D,E}$ (13), and the squared distance between the con- and pro-group $T_{ij}^D,T_{ij}^E,T_{ij}^{D,E}$ (14) for groups $D$ and $E$ are then obtained according to the given equations. The conflict with respect to alternative $A_j$ under attribute $G_i$ between the two groups $D$ and $E$ can now be represented by the between-group conflict index $d_{ij}^{D,E}$.

Definition 4 (between-group conflict index).

$$\begin{array}{}\text{(16)}& d_{ij}^{D,E}=\sqrt{\beta \left|\left(T_{ij}^{D,E}-\left(T_{ij}^D+T_{ij}^E\right)\right)-\text{(}\left(C^{D,E}-\left(C_{ij}^D+C_{ij}^E\right)\right)+\left(P_{ij}^{D,E}-\left(P_{ij}^D+P_{ij}^E\right)\right)\right|}\hspace{1em}\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\hspace{0.17em}\hspace{0.17em}\beta =\frac{\mathrm{1}}{\sum _{S_k\in S}{{\lambda }_k}^{\mathrm{2}}}\end{array}$$

The intuition behind the between-group conflict index is that conflict increases when two stakeholder groups’ con- and pro-groups have greater value differences between them. This can be noted by observing that $\mathrm{0}\le d_{ij}^{D,E}\le \mathrm{1}$, where $d_{ij}^{D,E}=\mathrm{1}$ when the two stakeholders groups have opposing preferences; for example, $C_{ij}^D=\mathrm{0}$, $C_{ij}^E=\mathrm{0}$, $C_{ij}^{D,E}=\mathrm{0}$, $P_{ij}^D=\mathrm{0}$, $P_{ij}^E=\mathrm{0}$, $P_{ij}^{D,E}=\mathrm{0}$, $T_{ij}^D=\mathrm{0}$, $T_{ij}^E=\mathrm{0}$, $T_{ij}^{D,E}=\mathrm{1}$; i.e., $(\mathrm{1}-(\mathrm{0}+\mathrm{0}))-((\mathrm{0}-(\mathrm{0}+\mathrm{0}))+(\mathrm{0}-(\mathrm{0}+\mathrm{0})))=\mathrm{1}$.

4.6. Example

Using the preference data in Table 2, we divide the group of stakeholders into two sets, $D=\{S_{\mathrm{1}},S_{\mathrm{2}},S_{\mathrm{3}},S_{\mathrm{4}},S_{\mathrm{5}}\}$ and $E=\{S_{\mathrm{6}},S_{\mathrm{7}},S_{\mathrm{8}},S_{\mathrm{9}},S_{\mathrm{10}}\}$, based on some demographic variable, e.g., age or sex. These sets are further divided into the con-group sets ${\mathit{S}}_{ij}^{D-}$ and ${\mathit{S}}_{ij}^{E-}$ and the pro-group sets ${\mathit{S}}_{ij}^{D+}$ and ${\mathit{S}}_{ik}^{E+}$. The conflict between the stakeholder groups is then given by the conflict index $d_{ij}^{D,E}$. The results of the calculations are presented in Table 6.

Table 6

The con- and pro-indices for both groups and the conflict between the groups.

	$$A_1$$	$$A_2$$	$$A_3$$	$$A_4$$	$$A_5$$
$$C_{i1}^{\mathit{D}}$$	0.0000	0.0000	0.0000	0.0000	0.0000
$$P_{i1}^{\mathit{D}}$$	0.0000	0.0002	0.0000	0.0008	0.0000
$$T_{i1}^{\mathit{D}}$$	0.0000	0.0002	0.0000	0.0008	0.0000
$$C_{i1}^{\mathit{E}}$$	0.0000	0.0002	0.0000	0.0008	0.0000
$$P_{i1}^{\mathit{E}}$$	0.0000	0.0000	0.0000	0.0000	0.0000
$$T_{i1}^{\mathit{E}}$$	0.0000	0.0002	0.0002	0.0008	0.0000
$$C_{i1}^{\mathit{D},\mathit{E}}$$	0.0000	0.0002	0.0000	0.0008	0.0000
$$P_{i1}^{\mathit{D},\mathit{E}}$$	0.0000	0.0002	0.0000	0.0008	0.0000
$$T_{i1}^{\mathit{D},\mathit{E}}$$	0.0000	0.0045	0.0002	0.0634	0.1000
$$d_{i1}^{\mathit{D},\mathit{E}}$$	0.0000	0.2000	0.0143	0.7857	1.0000

As in the within-group conflict index, the between-group conflict index range is $[\mathrm{0,1}]$. In this example we use the consensus properties defined in the within-group example in Section 4.4; see the conflict index intervals and the associated consensus properties in Table 3.

As seen in Table 6, the result for action $A_{\mathrm{1}}$ is $d_{\mathrm{1,1}}^{\mathit{D},\mathit{E}}=\mathrm{0}$, since there is no conflict between the groups. Table 2 shows that all stakeholders of both subsets $\mathit{D}$ ($S_{\mathrm{1}},\dots ,S_{\mathrm{5}}$) and $\mathit{E}$ ($S_{\mathrm{6}},\dots ,S_{\mathrm{10}}$) are members of the pro-group and that they all placed $A_{\mathrm{1}}$ on the same position as $A_{\alpha }$ in the cardinal ranking; i.e., no con-groups exist. The stakeholders in $\mathit{D}$ expressed that all other actions were better than $A_{\mathrm{1}}$, and stakeholders in $\mathit{E}$ expressed that all other actions were worse than action $A_{\mathrm{1}}$. Thus, when the ranking was normalized to proportional $[\mathrm{0,1}]$-value scale, the value for $A_{\mathrm{1}}$ was $v_{\mathrm{1,1}}^k$=0 for $S_{\mathrm{1}},\dots ,S_{\mathrm{5}}$, and $v_{\mathrm{1,1}}^k$=1 for $S_{\mathrm{6}},\dots ,S_{\mathrm{10}}$. Action $A_{\mathrm{2}}$ has $d_{\mathrm{1,2}}^{\mathit{D},\mathit{E}}=\mathrm{0.2}$, since the two stakeholder groups have similar preferences. Both groups estimated the value of the action to lie close to $A_{\alpha }$. Action $A_{\mathrm{3}}$ has $d_{\mathrm{1,3}}^{\mathit{D},\mathit{E}}=\mathrm{0.0143}$, note that this is because only stakeholder $S_{\mathrm{10}}$ estimates that the action is negative, and all other stakeholders estimate it to be equal to $A_{\alpha }$. Action $A_{\mathrm{4}}$ has $d_{\mathrm{1,4}}^{\mathit{D},\mathit{E}}=\mathrm{0.7857}$, since the two stakeholder groups have opposing preferences, as seen in Table 2. Action $A_{\mathrm{5}}$ has $d_{\mathrm{1,5}}^{\mathit{D},\mathit{E}}=\mathrm{1}$, since all members of $\mathit{D}$ estimate that the action is the best of all actions, and all members of $\mathit{E}$ estimate that the action is the worst of all actions. The results of the classification of the actions are presented in Table 7.

Table 7

The classification of the actions consensus properties.

Consensus Property	$$A_{\mathrm{1}}$$	$$A_{\mathrm{2}}$$	$$A_{\mathrm{3}}$$	$$A_{\mathrm{4}}$$	$$A_{\mathrm{5}}$$
Consensus aligned	$$\times$$		$$\times$$
Potentially controversial		$$\times$$
Controversial
Very Controversial				$$\times$$	$$\times$$

5. Application of CAR for Conflict Evaluations at Upplands Väsby Municipality

In a current research project, researchers from Stockholm University cooperate with the Royal Institute of Technology in developing models for public planning and decision processes in Swedish municipalities. We investigate how to increase expressiveness of the media used for communication between the parties in these processes. The focus here lies on developing tools that enrich this kind of communication. As a part of the project we conduct three case studies, of which one is in cooperation with civil servants and politicians at Upplands Väsby (UV), a municipality in the Stockholm region.

5.1. A Case Study

As reported in [13, Ch. 7], the politicians and civil servants wanted to get in-depth information regarding the citizens preferences regarding actions, which possibly could be implemented in the future. It was especially of interest to investigate if there were any actions that potentially could lead to citizen conflicts. To facilitate this, it was decided to use a web-based questionnaire (the Appendix) as a front-end for preference elicitation. The content of the questionnaire was developed together with civil servants at UV. The questionnaire consisted of four parts:

Part I consisted of ten questions regarding different “focus areas” (or criteria in an MCDA setting). Under each focus area, the respondents used an implementation of CAR for conflict evaluations to rank five actions; see Figure 3.

[figure omitted; refer to PDF]

Part II consisted of one question where the ten focus areas are given weights using an implementation of CAR; see Figure 4.

[figure omitted; refer to PDF]

Part III consisted of three questions where the respondents stated their preferences regarding two contradicting actions.

Part IV consisted of questions regarding demographic information.

An invitation letter containing a URL to the questionnaire was sent by mail to 10,000 citizens. The sample was chosen by conducting simple random sampling on a sampling frame consisting of 31,408 citizens. In the section below we present an analysis of the results from the eighth focus area, School. In total we received 939 answers; of these 465 were male, 456 female, 15 did not want to disclose their gender, and 3 selected other/agender. The analysis consists of two parts, the analysis of (i) the difference in preferences in the total population and (ii) the difference in preference between females and males.

5.1.1. Results

Five actions $A_{\mathrm{1}},\dots ,A_{\mathrm{5}}$ are evaluated under the focus area/attribute $G_{\mathrm{8}}$ School:

(i)

$A_{\mathrm{1}}$ reduces preschool child groups.

(ii)

$A_{\mathrm{2}}$ raises the quality of teaching.

(iii)

$A_{\mathrm{3}}$ increases professional development for schools and teachers.

(iv)

$A_{\mathrm{4}}$ increases modern information technology (IT) in education.

(v)

$A_{\mathrm{5}}$ involves caretakers more in school.

The respondents assess the affect associated with each action with regard to a do nothing alternative (current state). Note that this assessment may not be related to the cost of implementing the action, rather the feeling/affect related to the implementation. In the questionnaire implementation of the CAR for conflict evaluations (Figure 3) the do nothing alternative is represented by the tick mark located in the middle of the scale.

Analysis I: Conflict in the Population. In the first analysis we investigate the difference in preference in the total population. We used four consensus property labels to semantically describe the level of conflict. The within-group conflict index range $[\mathrm{0,1}]$ was divided into four subranges each representing one level of consensus; see Table 3. Note that the subranges and the consensus properties are defined by the authors for illustrative purposes.

The results of this analysis are presented in Table 8. The results show that all actions have a very low conflict index, indicating that the respondents have similar preferences. Actions $A_{\mathrm{1}}$ ($d^{{\mathit{S}}_{\mathrm{8,1}}}=\mathrm{0.0444}$), $A_{\mathrm{4}}$ ($d^{{\mathit{S}}_{\mathrm{8,1}}}=\mathrm{0.0440}$), and $A_{\mathrm{5}}$ ($d^{{\mathit{S}}_{\mathrm{8,5}}}=\mathrm{0.0412}$) have the largest conflict indices. Actions $A_{\mathrm{3}}$ and $A_{\mathrm{2}}$ have the lowest conflict indices of $d^{{\mathit{S}}_{\mathrm{8,3}}}=\mathrm{0.0244}$ and $d^{{\mathit{S}}_{\mathrm{8,2}}}=\mathrm{0.0197}$, respectively. The attribute properties are presented in Table 9, showing that all actions are consensus aligned.

Table 8

Analysis I: the number of members of the con- and pro-groups, the con- and pro-indices, and the conflict in the total population.

	$$A_1$$	$$A_2$$	$$A_3$$	$$A_4$$	$$A_5$$
$$|S_{\mathrm{8}j}^{-}|$$	88	11	24	133	120
$$|S_{\mathrm{8}j}^{+}|$$	851	928	915	806	819
$$C_{\mathrm{8}j}^{\mathit{S}}$$	$$1.04\cdot {10}^{-7}$$	$$2.74\cdot {10}^{-8}$$	$$7.32\cdot {10}^{-8}$$	$$1.64\cdot {10}^{-7}$$	$$1.52\cdot {10}^{-7}$$
$$P_{\mathrm{8}j}^{\mathit{S}}$$	$$4.91\cdot {10}^{-6}$$	$$5.57\cdot {10}^{-6}$$	$$5.54\cdot {10}^{-6}$$	$$4.87\cdot {10}^{-6}$$	$$4.68\cdot {10}^{-6}$$
$$T_{\mathrm{8}j}^{\mathit{S}}$$	$$7.12\cdot {10}^{-6}$$	$$6.01\cdot {10}^{-6}$$	$$6.25\cdot {10}^{-6}$$	$$7.09\cdot {10}^{-6}$$	$$6.64\cdot {10}^{-6}$$
$${\mathit{d}}_{\mathbf{8}\mathit{j}}^{\mathit{S}}$$	0.0444	0.0197	0.0244	0.0440	0.0412

Table 9

Analysis I: the classification of the actions consensus properties.

Consensus Property	$$A_{\mathrm{1}}$$	$$A_{\mathrm{2}}$$	$$A_{\mathrm{3}}$$	$$A_{\mathrm{4}}$$	$$A_{\mathrm{5}}$$
Consensus aligned	$$\times$$	$$\times$$	$$\times$$	$$\times$$	$$\times$$
Potentially controversial
Controversial
Very Controversial

Analysis II: The Difference in Preference between Females and Males. In the second analysis we investigate the difference in preference between females and males. We use the same consensus property labels as in the first analysis to semantically describe the level of conflict. The between-group conflict index range $[\mathrm{0,1}]$ was divided into four subranges each representing one level of consensus; see Table 3. Note that the consensus properties are defined by the authors for illustrative purposes.

The results of the calculations are presented in Table 10. All actions have very low conflict indices, which also is reflected in the attribute properties (Table 11), where all actions are consensus aligned. The action with the highest conflict index is $A_{\mathrm{1}}$ ($d^{{\mathit{S}}_{\mathrm{8,1}}}=\mathrm{0.0107}$), followed by $A_{\mathrm{4}}$ ($d^{{\mathit{S}}_{\mathrm{8,4}}}=\mathrm{0.0052}$), $A_{\mathrm{3}}$ ($d^{{\mathit{S}}_{\mathrm{8,3}}}=\mathrm{0.0027}$), $A_{\mathrm{5}}$ ($d^{{\mathit{S}}_{\mathrm{8,5}}}=\mathrm{0.0019}$), and $A_{\mathrm{2}}$ ($d^{{\mathit{S}}_{\mathrm{8,2}}}=\mathrm{0.0005}$). The difference in value between the actions’ conflict indices is small since both groups have very similar preferences.

Table 10

Analysis II: the number of members of the con- and pro-groups, the con- and pro-indices, and the conflict between females and males.

	$$A_1$$	$$A_2$$	$$A_3$$	$$A_4$$	$$A_5$$
$$|S_{8j}^{D-}|$$	25	6	10	60	57
$$|S_{8j}^{D+}|$$	431	450	446	396	399
$$|S_{8j}^{E-}|$$	60	5	13	69	61
$$|S_{8j}^{E+}|$$	405	460	452	396	404
$$C_{8j}^{\mathit{D}}$$	$$\mathrm{3.60}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$	$$\mathrm{1.29}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$	$$\mathrm{6.00}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$	$$\mathrm{6.45}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$	$$\mathrm{6.87}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$
$$P_{8j}^{\mathit{D}}$$	$$\mathrm{2.39}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{2.70}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{2.68}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{2.07}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{2.31}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$
$$T_{8j}^{\mathit{D}}$$	$$\mathrm{3.20}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{2.94}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{3.07}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{2.95}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{3.20}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$
$$C_{8j}^{\mathit{E}}$$	$$\mathrm{6.82}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$	$$\mathrm{1.56}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$	$$\mathrm{6.20}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{9}}$$	$$\mathrm{9.61}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$	$$\mathrm{8.70}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$
$$P_{8j}^{\mathit{E}}$$	$$\mathrm{2.57}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{3.02}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{3.03}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{2.85}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{2.52}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$
$$T_{8j}^{\mathit{E}}$$	$$\mathrm{3.86}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{3.24}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{3.33}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{4.23}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{3.62}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$
$$C_{8j}^{\mathit{D},\mathit{E}}$$	$$\mathrm{1.05}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{7}}$$	$$\mathrm{2.85}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$	$$\mathrm{7.39}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{8}}$$	$$\mathrm{1.61}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{7}}$$	$$\mathrm{1.57}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{7}}$$
$$P_{8j}^{\mathit{D},\mathit{E}}$$	$$\mathrm{5.05}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{5.72}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{5.71}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{4.99}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{4.83}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$
$$T_{8j}^{\mathit{D},\mathit{E}}$$	$$\mathrm{7.27}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{6.18}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{6.41}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{7.22}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$	$$\mathrm{6.82}\cdot \mathrm{1}{\mathrm{0}}^{-\mathrm{6}}$$
$${\mathit{d}}_{\mathit{8}\mathit{j}}^{\mathit{D},\mathit{E}}$$	0.0107	0.0005	0.0027	0.0052	0.0019

Table 11

Analysis II: the classification of the actions consensus properties.

Consensus Property	$$A_{\mathrm{1}}$$	$$A_{\mathrm{2}}$$	$$A_{\mathrm{3}}$$	$$A_{\mathrm{4}}$$	$$A_{\mathrm{5}}$$
Consensus aligned	$$\times$$	$$\times$$	$$\times$$	$$\times$$	$$\times$$
Potentially controversial
Controversial
Very Controversial

6. Concluding Remarks

Public decision problems can be complex. They involve multiple actions and multiple stakeholders which may have conflicting preferences with regard to the actions. A problem in such situations is that these opposing preferences might lead to conflicts between stakeholders, which may lead to delays in the decision process. An approach that can be used to increase the understanding of the stakeholders’ opinions is to allow them to state their opinions, e.g., by using a web-based questionnaire.

In this paper, we showed how the CAR method can be applied, enabling respondents to state negative or positive preferences with regard to an alternative’s performance relative to a do nothing alternative. We applied CAR for conflict evaluations in a case study conducted in cooperation with Upplands Väsby municipality. The citizens’ preferences were elicited by a web-based questionnaire that used an implementation of the method. We showed how the method can be used to highlight conflict between and within different stakeholders groups and how the conflict can be conceptualized into semantic attribute properties. The results of such an analysis can aid decision-makers in the process of making well-informed decisions by clarifying the actions that can be conflict-prone.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

This research is partly funded by The Swedish Research Council Formas, grant 2011-3313-20412-31.

Appendix

Appendix

The Questionnaire

Part I. What Should Upplands Väsby Focus on in the Future?

(1)

Parks and greenbelts

(1a)

Preserve existing larger greenbelts

(1b)

Build parks in existing urban districts

(1c)

Build homes near greenbelts

(1d)

Renovate existing parks

(1e)

Improve accessibility to major greenbelts

(2)

Diversity in housing supply

(2a)

Offer more housing types

(2b)

Offer more apartment sizes

(2c)

Offer small-scale land ownership

(2d)

Preserve the conceptual foundations of the buildings from the 1970s

(2e)

Offer more housing near the water

(3)

Vitalize common places

(3a)

Mix different types of traffic

(3b)

Place parking along the streets

(3c)

Turn entrances to streets

(3d)

Place public locales in transparent ground floors

(3e)

Secure parking solutions under houses

(4)

Communications

(4a)

Pair the new streets with existing ones to strengthen the connection to the adjacent neighborhoods and reduce the barriers that the main roads pose.

(4b)

Improve communications at night between various parts of the municipality

(4c)

Improve communications to and from Uppsala

(4d)

Improve the north-south and east-west routes through a fine-mesh and well-integrated metropolitan area networks.

(4e)

Improve communications to and from downtown Stockholm

(5)

Culture and recreation

(5a)

Expand the range of cultural sports and recreational activities

(5b)

Create better opportunities for festivals and concerts

(5c)

Create more opportunities for outdoor sports

(5d)

Create outdoors marketplaces

(5e)

Provide municipal grants for cultural and recreational projects

(6)

Education

(6a)

Renovate older schools

(6b)

Build new schools

(6c)

Improve the physical environment of schoolyards

(6d)

Improve the quality of primary education

(6e)

Improve the quality of secondary education

(7)

Care

(7a)

Increase cultural and recreational activities for the elderly

(7b)

Increase cultural and recreational activities for children and young people

(7c)

Improve care for the elderly in the municipality

(7d)

Increase youth centres and field assistants

(7e)

Reduce preschool child groups

(8)

School

(8a)

Reduce preschool child groups

(8b)

Raise the quality of teaching

(8c)

Increase professional development for schools and teachers

(8d)

Increase modern information technology (IT) in education

(8e)

Involve caretakers more in school

(9)

Safety

(9a)

Increase safety around the station area

(9b)

Increase police presence in central Väsby

(9c)

Improve the lighting in the centre of Väsby

(9d)

Narrow opening hours for alcohol outlets in central Väsby

(9e)

Extend the opening hours of shops in the city center

(10)

Sustainable development

(10a)

Reduce energy consumption

(10b)

Reduce transport and sound pollution

(10c)

Increase climate change adaptation and recycling

(10d)

Prioritize environmentally friendly transport modes (walking, cycling, public transport)

(10e)

Reducing environmental toxins and hazardous chemicals in nature

Part II. How Important Is Each Focus Area?

(11)

Weighting

(1)

Parks and greenbelts

(2)

Diversity in housing supply

(3)

Vitalize common places

(4)

Communication

(5)

Culture and recreation

(6)

Education

(7)

Care

(8)

School

(9)

Safety

(10)

Sustainable development

Part III. Contradictions

(12)

Water or housing

(12a)

Build homes near water

(12b)

Preserving nature and shorelines intact

(13)

Services or green areas

(13a)

Densify the city centre and increase the range of services

(13b)

Preserve green areas

(14)

Regional centre or smaller urban areas

(14a)

Develop central Upplands Väsby

(14b)

Develop smaller urban areas

Part IV. Your Background

(i)

Where do you live? (This question consisted of 41 residential areas to choose among)

(ii)

What is your highest level of education?

(a)

Not finished elementary school or equivalent compulsory school

(b)

High school or equivalent compulsory school

(c)

High school, folk high-school or equivalent

(d)

College/university

(e)

Other post-secondary education

(f)

Postgraduate

(iii)

What is your main occupation?

(a)

Employed

(b)

Self-employed

(c)

Student

(d)

Pensioner/retired

(e)

Sickness or activity compensation

(f)

Long term sick leave (more than 3 months)

(g)

Leave of absence or parental leave

(h)

Job seeker or in labour market activity

(i)

Homemaker

(j)

Other

(iv)

How long have you lived in Upplands Väsby?

(a)

0-4 years

(b)

5-9 years

(c)

10 years or more

(v)

How old are you?

(a)

0-14

(b)

15-24

(c)

25-34

(d)

35-44

(e)

45-54

(f)

55-64

(g)

65+

(vi)

What is your gender?

(a)

Female

(b)

Male

(c)

Other/agender

(d)

Do not want to disclose

References

[1] S. French, D. Rios Insua, F. Ruggeri, Decision Analysis, vol. 4 no. 4, pp. 211-226, DOI: 10.1287/deca.1070.0098, 2007.

[2] C. Bayley, S. French, "Designing a participatory process for stakeholder involvement in a societal decision," Group Decision and Negotiation, vol. 17 no. 3, pp. 195-210, DOI: 10.1007/s10726-007-9076-8, 2007.

[3] M. Riabacke, J. Åström, Å. Grönlund, "eParticipation galore? extending multi-criteria decision analysis to the public," International Journal of Public Information Systems, vol. 70 no. 2, 2011.

[4] M. Danielson, L. Ekenberg, J. Idefeldt, A. Larsson, "Using a software tool for public decision analysis: the case of nacka municipality," Decision Analysis, vol. 4 no. 2, pp. 76-90, DOI: 10.1287/deca.1070.0088, 2007.

[5] M. Danielson, L. Ekenberg, A. Ekengren, T. Hökby, J. Lidén, "Decision process support for participatory democracy," Journal of Multi-Criteria Decision Analysis, vol. 15 no. 1-2, pp. 15-30, DOI: 10.1002/mcda.406, 2008.

[6] K. Hansson, G. Cars, M. Danielson, L. Ekenberg, A. Larsson, "Diversity and public decision making. in world academy of science, engineering and Technology 71," Proceedings of the 2012 International Conference on e-Democracy and Open Government, pp. 1678-1683, .

[7] B. Enserink, "A quick scan for infrastructure planning: Screening alternatives through interactive stakeholder analysis," Impact Assessment and Project Appraisal, vol. 18 no. 1, pp. 15-22, DOI: 10.3152/147154600781767628, 2000.

[8] M. S. Reed, A. Graves, N. Dandy, H. Posthumus, K. Hubacek, J. Morris, C. Prell, C. H. Quinn, L. C. Stringer, "Who's in and why? A typology of stakeholder analysis methods for natural resource management," Journal of Environmental Management, vol. 90 no. 5, pp. 1933-1949, DOI: 10.1016/j.jenvman.2009.01.001, 2009.

[9] R. Efremov, D. R. Insua, A. Lotov, "A framework for participatory decision support using Pareto frontier visualization, goal identification and arbitration," European Journal of Operational Research, vol. 199 no. 2, pp. 459-467, DOI: 10.1016/j.ejor.2008.10.034, 2009.

[10] M. Danielson, L. Ekenberg, "The CAR Method for Using Preference Strength in Multi-criteria Decision Making," Group Decision and Negotiation, vol. 25 no. 4, pp. 775-797, DOI: 10.1007/s10726-015-9460-8, 2016.

[11] A. C. Rencher, Method of Multivariate Analysis,DOI: 10.1002/0471271357, 2003.

[12] C. Bana e Costa, "The use of multi-criteria decision analysis to support the search for less conflicting policy options in a multi-actor context: Case study," Journal of Multi-Criteria Decision Analysis, vol. 10 no. 2, pp. 111-125, DOI: 10.1002/mcda.292, 2001.

[13] L. Ekenberg, K. Hansson, M. Danielson, G. Cars, L. In de Betou, J. Buurman, M. Ernst, T. Fasth, R. Forsberg, J. Gustafsson Fürst, K. E. Hansson, P. Karlström, F. N. Kivunike, A. Larsson, T. Liljenberg, H. Liljenström, A. Marincea, A. Mihai, M. Riabacke, W. Sauter, U. Svedin, M. Thompson, F. Tusubira, H. Verhagen, M. Wrange, Deliberation, Representation, Equity: Research Approaches, Tools and Algorithms for Participatory Processes,DOI: 10.11647/OBP.0108, 2017.

[14] E. Vilkkumaa, A. Salo, J. Liesiö, "Multicriteria Portfolio Modeling for the Development of Shared Action Agendas," Group Decision and Negotiation, vol. 23 no. 1, pp. 49-70, DOI: 10.1007/s10726-012-9328-0, 2013.

[15] M. Riabacke, A Prescriptive Approach to Eliciting Decision Information [Ph.D. thesis], 2012.

[16] E. Jacquet-Lagrèze, Y. Siskos, "Preference disaggregation: 20 Years of MCDA experience," European Journal of Operational Research, vol. 130 no. 2, pp. 233-245, DOI: 10.1016/S0377-2217(00)00035-7, 2001.

[17] L. Dias, V. Mousseau, J. Figueira, J. Clímaco, "An aggregation/disaggregation approach to obtain robust conclusions with ELECTRE TRI," European Journal of Operational Research, vol. 138 no. 2, pp. 332-348, DOI: 10.1016/S0377-2217(01)00250-8, 2002.

[18] M. Ghaderi, F. Ruiz, N. Agell, "A linear programming approach for learning non-monotonic additive value functions in multiple criteria decision aiding," European Journal of Operational Research, vol. 259 no. 3, pp. 1073-1084, DOI: 10.1016/j.ejor.2016.11.038, 2017.

[19] S. Greco, V. Mousseau, R. Słowiński, "Ordinal regression revisited: multiple criteria ranking using a set of additive value functions," European Journal of Operational Research, vol. 191 no. 2, pp. 416-435, DOI: 10.1016/j.ejor.2007.08.013, 2008.

[20] J. S. Dyer, R. K. Sarin, "Measurable multiattribute value functions," Operations Research. The Journal of the Operations Research Society of America, vol. 27 no. 4, pp. 810-822, DOI: 10.1287/opre.27.4.810, 1979.

[21] J. S. Dyer, R. K. Sarin, "Group preference aggregation rules based on strength of preference," Management Science, vol. 25 no. 9, pp. 822-832 (1980), DOI: 10.1287/mnsc.25.9.822, 1979.

[22] F. Eisenführ, M. Weber, T. Langer, Rational Decision Making,DOI: 10.1007/978-3-642-02851-9, 2010.

[23] M. Riabacke, M. Danielson, L. Ekenberg, "State-of-the-art prescriptive criteria weight elicitation," Advances in Decision Sciences, vol. 2012, 2012.

[24] W. G. Stillwell, D. A. Seaver, W. Edwards, "A comparison of weight approximation techniques in multiattribute utility decision making," Organizational Behavior and Human Decision Processes, vol. 28 no. 1, pp. 62-77, DOI: 10.1016/0030-5073(81)90015-5, 1981.

[25] F. H. Barron, "Selecting a best multiattribute alternative with partial information about attribute weights," Acta Psychologica, vol. 80 no. 1–3, pp. 91-103, DOI: 10.1016/0001-6918(92)90042-c, 1992.

[26] F. H. Barron, B. E. Barrett, "The efficacy of SMARTER—simple multi-attribute rating technique extended to ranking," Acta Psychologica, vol. 93 no. 1–3, pp. 23-36, DOI: 10.1016/0001-6918(96)00010-8, 1996.

[27] C. W. Kirkwood, R. K. Sarin, "Ranking with partial information: a method and an application," Operations Research, vol. 33 no. 1, pp. 38-48, DOI: 10.1287/opre.33.1.38, 1985.

[28] F. H. Barron, B. E. Barrett, "Decision quality using ranked attribute weights," Management Science, vol. 42 no. 11, pp. 1515-1523, DOI: 10.1287/mnsc.42.11.1515, 1996.

[29] J. Jia, G. W. Fischer, J. S. Dyer, "Attribute weighting methods and decision quality in the presence of response error: A simulation study," Journal of Behavioral Decision Making, vol. 11 no. 2, pp. 85-105, DOI: 10.1002/(SICI)1099-0771(199806)11:2<85::AID-BDM282>3.0.CO;2-K, 1998.

[30] M. Danielson, L. Ekenberg, Y. He, "Augmenting ordinal methods of attribute weight approximation," Decision Analysis, vol. 11 no. 1, pp. 21-26, DOI: 10.1287/deca.2013.0289, 2014.

[31] T. L. Saaty, L. G. Vargas, "How to make a decision," Multiple Criteria Decision Analysis: State of the Art Surveys,DOI: 10.1007/b100605, 2012.

[32] C. Bana e Costa, J.-C. Vansnick, "MACBETH—an interactive path towards the construction of cardinal value functions," International Transactions in Operational Research, vol. 1 no. 4, pp. 489-500, DOI: 10.1016/0969-6016(94)90010-8, 1994.

[33] C. A. Bana e Costa, J. De Corte, J. Vansnick, "On the Mathematical Foundation of MACBETH," Multiple Criteria Decision Analysis: State of the Art Surveys, vol. 78, pp. 409-437, DOI: 10.1007/0-387-23081-5_10, 2005.

[34] A. Larsson, M. Riabacke, M. Danielson, L. Ekenberg, "Cardinal and rank ordering of criteria - Addressing prescription within weight elicitation," International Journal of Information Technology & Decision Making, vol. 14 no. 6, pp. 1299-1330, DOI: 10.1142/S021962201450059X, 2015.

[35] J. G. Jaspersen, G. Montibeller, "Probability Elicitation Under Severe Time Pressure: A Rank-Based Method," Risk Analysis, vol. 35 no. 7, pp. 1317-1335, DOI: 10.1111/risa.12357, 2015.

[36] R. Lahdelma, P. Salminen, J. Hokkanen, "Using multicriteria methods in environmental planning and management," Journal of Environmental Management, vol. 26 no. 6, pp. 595-605, DOI: 10.1007/s002670010118, 2000.

[37] R. T. Clemen, J. E. Smith, "On the Choice of Baselines in Multiattribute Portfolio Analysis: A Cautionary Note," Decision Analysis, vol. 6 no. 4, pp. 256-262, DOI: 10.1287/deca.1090.0158, 2009.

[38] E. P. Torrance, "Group Decision-Making and Disagreement," Social Forces, vol. 35 no. 4, pp. 314-318, DOI: 10.2307/2573319, 1957.