Multidimensional Evaluation of Consumers’ Shopping Risks under Live-Streaming Commerce

Abstract

As a new e-commerce model, live-streaming commerce enhances consumers’ shopping experiences by providing deeply involved online interaction. However, in live-streaming commerce, consumers are also faced with many shopping risks, such as fake products, poor after-sales service, etc. Therefore, we propose an analysis framework to evaluate consumers’ shopping risks on live-streaming commerce platforms. In our framework, we first construct a multidimensional consumer shopping risk evaluation index system by considering different stakeholders involved in live-streaming commerce. Then, we assess consumer shopping risks based on an intuitionistic fuzzy analytic hierarchy process and cloud model. Our framework is applied to evaluate consumers’ shopping risks on four typical live streaming commerce platforms in China, i.e., Taobao, Douyin, Kuaishou, and JD.com. Our research results provide decision support for different parties involved in live-streaming commerce and thus promote the sustainable development of the live-streaming commerce industry.

1. Introduction

In recent years, live-streaming commerce has become popular in China. Live-streaming commerce is a new e-commerce model in which web hosts sell merchandise while live streaming online. Live-streaming commerce is becoming a new driving force for e-commerce consumption growth, and almost all major e-commerce platforms in China have enabled the live-streaming function. As of December 2022, the number of users of China’s e-commerce live broadcast had reached 515 million, accounting for 48.2% of the total number of Chinese netizens [1]. Ordinary people, Internet celebrities, celebrities, and even government officials have appeared in various webcast rooms, selling various products to the audience. For example, Li Jiaqi’s live broadcast room achieved a transaction volume of 2.9 billion USD on just one night, 24 October 2022 [2].

While consumers enjoy the benefits of instant interaction and high engagement in live streaming commerce, they also face many shopping risks, such as poor after-sales service, slow logistics speeds, fake products, fraudulent orders, etc. Therefore, it is necessary to evaluate consumers’ shopping risks on live-streaming commerce platforms, which will help promote the sustainable development of live streaming commerce and improve consumer satisfaction.

Since e-commerce platforms use Internet technology to conduct transactions, which involve a large quantity of user data and transaction information, many researchers focus on the network security risk assessment of e-commerce. Karoui [3] proposed a risk assessment analysis framework based on a set of reversible indicators to assess the risks of some distributed denial of service attacks for an e-commerce enterprise. Zhang et al. [4] established a multilevel index system and propose a security assessment model for e-commerce based on AHP and the DS theory of evidence to assess the e-commerce security risk.

In addition to network security risks, e-commerce platforms also face many other risks, such as privacy risks, commodity risks, and financial risks. Dinev and Hart [5] studied the impact of privacy security risks and consumer beliefs on e-commerce transactions. Song et al. [6] proposed a risk assessment method using text mining and fuzzy rule-based reasoning to quantitatively and semi-automatically assess the commodity risk of cross-border e-commerce. Liu et al. [7] constructed an evaluation model based on machine learning methods to identify the product quality risk of the e-commerce platform. To consider risks more comprehensively, Mou et al. [8] conducted a more comprehensive study on the risks of cross-border e-commerce from the perspectives of customer duties risk, confiscation risk, delivery risk, financial risk, and privacy risk.

The above-mentioned studies on e-commerce risks mainly focus on traditional e-commerce. There are few studies on the risks of live streaming commerce. Live-streaming commerce is a means to enhance merchants’ electronic word of mouth (E-WOM) and promote product sales. Merchants are willing to utilize various means to enhance their E-WOM [9,10,11], which in turn enhances consumers’ purchase intentions [12]. Therefore, existing live streaming commerce studies mainly focus on the impact of live streaming commerce on consumer purchase intentions. From the perspective of consumers’ trust, Dong et al. [13] showed that live streaming enhanced their willingness to purchase. Similarly, Lu and Chen [14] analyzed the impact of live broadcasters on consumers’ purchase intentions from the perspective of reducing product uncertainty and cultivating trust for consumers. There are also some studies [15,16,17] on how live e-commerce can increase consumers’ purchase intentions by increasing consumer participation. Wang et al. [18] focused on platform-based information processing mechanisms to study how to transform customers’ help-seeking propensity into purchase intentions.

There are also some studies that concentrate on the impact of live-streaming commerce on merchant operations. Gong et al. [19] find that the standardization and quality of products will affect the effect of live streaming commerce on online merchants’ profits. Wongkitrungrueng and Assarut [20] investigated how customers’ perceived value of live-streaming commerce influence their trust and engagement with merchants. Their research revealed that live streaming could help merchants increase sales and loyalty. Zhang et al. [21] studied the impact of opening a live-streaming commerce channel on the company’s existing sales mode.

In summary, the existing studies on live streaming commerce mainly focus on the merchants’ perspective as well as factors affecting consumers’ purchasing intentions. The consumer’s risk assessment of live streaming commerce receives little attention. To fill this gap, we investigated the research question of how to effectively evaluate consumers’ shopping risks on live-streaming commerce platforms. To achieve this goal, we propose an analysis framework to evaluate consumers’ shopping risks on live-streaming commerce platforms. Our results indicate that in order to effectively reduce consumers’ shopping risks in live-streaming commerce, we should first focus on product-related risk factors. Our research results provide practical reference for researchers as well as decision support for different parties involved in live-streaming commerce.

Our main contributions are as follows: (a) We construct a multidimensional consumer shopping risk evaluation index system by considering different stakeholders involved in live-streaming commerce. (b) We assess consumers’ shopping risk based on an intuitionistic fuzzy analytic hierarchy process (IFAHP) and cloud model. (c) To validate the effectiveness and usefulness of our analytical framework, we apply the framework to evaluate consumers’ shopping risks on four typical live-streaming commerce platforms in China, i.e., Taobao, Douyin, Kuaishou, and JD.com.

The remainder of this paper is organized as follows. In Section 2, we propose a consumers’ shopping risk evaluation framework in live streaming commerce. Section 3 constructs a multidimensional consumer shopping risk evaluation index system based on the proposed framework. Consumers’ shopping risks on four typical live streaming commerce platforms in China are evaluated in Section 4. The theoretical and managerial implications of our research are discussed in Section 5. The Section 6 concludes the paper.

2. Background

By analyzing the stakeholders and their interrelationships in live-streaming commerce, we constructed an analysis framework for evaluating consumers’ shopping risks (Figure 1). Our subsequent research is based on this framework.

The stakeholders in live-streaming commerce involve consumers, merchants, live streamers, platforms, payment service providers, and logistics service providers. The live-streaming commerce platform provides a product sales channel for merchants and reviews the qualifications of merchants and live streamers. The merchants entrust live streamers with promoting products on the platform. The live streamers sell products to consumers online through the platform’s live streaming rooms. The payment and logistics service providers provide payment and logistics support for live streaming commerce.

For consumers, the process of live-streaming commerce is as follows. Consumers register as users on a live-streaming commerce platform and watch hosts introduce products. Consumers can directly click on the purchase link on the platform to place an order. After a successful payment, the merchant ships products through a logistics service provider. After receiving the products, consumers may also experience after-sales services such as returns and exchanges.

In each link of the above process, consumers may face different degrees of risk. In order to effectively evaluate these risks, our framework first constructs a multidimensional risk index system. The system establishes a comprehensive subdivision index of consumer shopping risk based on the dimensions of different stakeholders. Then, based on IFAHP and cloud model, the shopping risks faced by consumers are quantified.

Based on our analysis framework, we verify the following hypotheses: (1) As live-streaming commerce is an emerging e-commerce mode [19], consumers’ shopping risks in this mode mainly come from the commodity dimension [22], and live streamers, platforms, payment service providers, and logistics service providers have a certain impact on the risk. (2) The platform’s E-WOM affects consumers’ perceived risk [23]. The risk level of Taobao and JD.com is lower than that of Kuaishou and Douyin because Taobao and JD.com have better E-WOM as established e-commerce platforms in China.

3. Index System for Consumers’ Shopping Risk Evaluation

Our proposed index system for evaluating consumers’ shopping risks in live-streaming commerce contains 5 first-level indicators (commodity, live streamer, platform, payment, and logistics) and 18 second-level indicators (Figure 2). The first-level indicators are identified based on the stakeholders discussed in Section 2.

The commodity dimension risk was measured by the following second-level indicators: (1) Counterfeiting and inferior quality (B1): merchants counterfeit products by using other brand names or inferior and harmful materials that endanger consumers’ safety [22,24]. (2) Unreasonable price (B2): for example, the price of products is claimed to be “the lowest price on the whole network”, but it is a cover-up [25]. Merchants will increase the price of products under the influence of internet celebrities to maintain a certain profit. (3) No products received (B3): merchants do not ship products due to reasons such as setting wrong shopping link prices or making false or incorrect shipments, making it impossible for consumers to receive products [26]. (4) Refusing to return or exchange products (B4): merchants unreasonably refuse to return or exchange products and evade their responsibilities, which is not conducive to consumers defending their legitimate rights and interests [26,27].

The live-streamer dimension risk is usually caused by the live-streaming platform’s failure to strengthen the audit management of the live streamers. We measured this risk based on the following second-level indicators: (1) Poor professional ability (B5): the live streamers’ unfamiliarity with relevant professional knowledge and poor ability to screen products make it difficult for consumers to obtain valuable information about products [28]. (2) Lack of credit (B6): the live streamers use their position of information superiority to exaggerate the efficacy of products and induce consumers to buy products, or they create false data illusions by buying fans and comments [29]. (3) Shirking of responsibility after sales (B7): the live streamers shirk their propaganda and legal responsibility for selling products in the independent sales mode, or they shirk the joint responsibility that should be shared with the advertiser in the entrusted sales mode.

The platform dimension risk comes from the platform’s own functions, and its second-level indicators were as follows: (1) Unfriendly interface design (B8): it reflects consumers’ overall uncomfortable perception of page layout and visual effects, the difficulty in platform operation, and not being easy to find products or services needed [30]. (2) Unreasonable function settings (B9): the platform function defect causes page lag and unresponsiveness, reducing the consumers’ shopping experience [31,32], or the platform function is incomplete, such as on some live-streaming commerce platforms that cannot provide the replay function of live streaming content. (3) Poor service feedback (B10): the communication channels set by the live-streaming commerce platform are few, and the willingness, enthusiasm, and speed with which they handle consumer problems cannot satisfy consumers [33]. (4) Weak platform personal privacy protection (B11): the platform’s privacy protection system is missing or shares consumer personal information and even consumption habits without the consumer’s permission [22].

The payment dimension risk is mainly caused by the flawed payment system and improper risk prevention measures. This risk was measured based on the following second-level indicators: (1) Private payment method (B12): consumers are induced to pay through personal Alipay, WeChat, and other methods without going through the shopping link of the live-streaming commerce platform. (2) Financial loss caused by payment (B13): it is reflected in the financial loss caused by the insecurity of the network or payment platform when consumers pay, such as a loss of payment and the theft of bank cards [34]. (3) Payment interruption or cancellation (B14): consumers’ payment behavior is erroneous due to internal staff errors or necessary system downtime maintenance, force majeure, and other factors during payment [35]. (4) Payment data tampering or theft (B15): some high-risk attacks will cause consumers’ payment data to be tampered with, and accounts and passwords can even be stolen [34].

The logistical risk is usually caused by poor delivery service quality and imperfect logistics infrastructure. The following second-level indicators were used to measure this risk: (1) Slow delivery speed (B16): the time between delivery and receipt is so long that products are not delivered to the consumer on time [36,37]. (2) Damage and loss of products (B17): Due to asymmetric information, consumers cannot judge the quality of logistics services. Logistics transportation could cause damage or loss of products, which cannot guarantee their integrity [22,25]. (3) High logistics service costs (B18): the delivery cost of logistics is too high compared to the value of the products themselves, or the freight for returns and exchanges due to transaction disputes may be borne by consumers [37].

4. Multidimensional Risk Evaluation Based on IFAHP and Cloud Model

Based on the index system presented in the previous section, this section proposes a multidimensional risk evaluation method that combines the IFAHP and a cloud model to quantitatively evaluate consumers’ shopping risk. First, we chose the top four live-streaming commerce platforms as our evaluation objects: Taobao, Douyin, Kuaishou, and JD.com [38].

Then, we found 20 typical live-streaming commerce users to provide input information for the subsequent risk evaluation. These users had the following characteristics:

• They had shopping experience on the above four live-streaming commerce platforms;

• They had at least two years of online shopping experience on these platforms;

• They watched live streaming on these platforms for at least 3 h per week;

• They shopped on these platforms at least three times per month;

• They bought no less than three kinds of goods online every month.

Finally, typical users were asked to compare the importance of different indicators and rate the relative importance of any two indicators of the same category on a 9-point scale. The indicator weights were determined through IFAHP (Section 4.1). Based on the cloud model and the scoring data of the minimum and maximum values of the risk dimensions faced by typical users, the shopping risk faced by consumers on different platforms was comprehensively quantified (Section 4.2).

4.1. Determining Indicator Weights Based on IFAHP

Different from the AHP and FAHP, the IFAHP simultaneously considers the importance, unimportance, and hesitation of each indicator, which better reflect the opinions and judgment tendencies of the typical users. Moreover, the revising algorithm in the consistency test of the IFAHP helps the typical user make decisions to save time without losing much original information. Taking the first-level indicators as an example, we determined the indicator weights based on the IFAHP in the following steps.

Step 0: We use the intuitionistic fuzzy AHP to transform the collected data into a single user’s intuitionistic fuzzy matrix ${\left(p_{cij}\right)}_{n\times n}$, where $p_{cij}$ represents the importance degree of indicator $i$ relative to indicator $j$ evaluated by user $c$.

Step 1: We construct the intuitionistic fuzzy judgment matrix $R_A$ for the first-level indicators. $R_A={\left(r_{ij}\right)}_{n\times n}$, where $r_{ij}=\left(u_{ij},v_{ij}\right)$, $i,j=1,\dots ,n$, $n$ is the number of first-level indicators. $u_{ij}$ denotes the degree of affiliation, i.e., the degree of importance of indicator $i$ over indicator $j$; $v_{ij}$ denotes the degree of nonaffiliation, i.e., the degree of the unimportance of indicator $i$ over indicator $j$. $u_{ij}$ and $v_{ij}$ are obtained by using the IFA operator to gather the scoring information of each typical user [38]. $u_{ij}$ and $v_{ij}$ are calculated by the following equation:

(1)$u_{ij}=\{\begin{array}{cc}1-{\left({\displaystyle {\displaystyle \prod }_{n=1}^{20}}\left(1-p_{cij}\right)\right)}^{\frac{1}{20}}& , i<j\hfill \\ 0.5& ,i=j\hfill \\ {\left({\displaystyle {\displaystyle \prod }_{n=1}^{20}}{p}_{cij}\right)}^{\frac{1}{20}}& ,i>j\hfill \end{array}$

(2)$v_{ij}=u_{ji}$

The degree of hesitation ${\pi }_{ij}=1-u_{ij}-v_{ij}$. Thus, we have

$R_A=\left[\begin{array}{ccccc}\left(0.5000,0.5000\right)& \left(0.7782,0.1670\right)& \left(0.7166,0.1937\right)& \left(0.6717,0.2329\right)& \left(0.6020,0.2497\right)\\ \left(0.1670,0.7782\right)& \left(0.5000,0.5000\right)& \left(0.7208,0.2124\right)& \left(0.6687,0.2400\right)& \left(0.6442,0.2376\right)\\ \left(0.1937,0.7166\right)& \left(0.2124,0.7208\right)& \left(0.5000,0.5000\right)& \left(0.6210,0.2587\right)& \left(0.5904,0.2670\right)\\ \left(0.2329,0.6717\right)& \left(0.2400,0.6687\right)& \left(0.2587,0.6210\right)& \left(0.5000,0.5000\right)& \left(0.6001,0.2510\right)\\ \left(0.2497,0.6020\right)& \left(0.2376,0.6442\right)& \left(0.2670,0.5904\right)& \left(0.2510,0.6001\right)& \left(0.5000,0.5000\right)\end{array}\right]$

Step 2: We perform a consistency test for the intuitionistic fuzzy judgment matrix $R_A$. Before the consistency test, $R_A$ needs to be converted into a multiplicative, consistent, intuitionistic fuzzy judgment matrix ${\overline{R}}_A={\left({\overline{r}}_{ij}\right)}_{nxn}$, where ${\overline{r}}_{ij}$ is the multiplicative, consistent, intuitionistic preference of indicator $i$ over indicator $j$ [39]. Moreover, we have

${\overline{R}}_A=\left[\begin{array}{ccccc}\left(0.5000,0.5000\right)& \left(0.7782,0.1670\right)& \left(0.9006,0.0513\right)& \left(0.8441,0.0679\right)& \left(0.8055,0.0760\right)\\ \left(0.1670,0.7782\right)& \left(0.5000,0.5000\right)& \left(0.7208,0.2124\right)& \left(0.8088,0.0860\right)& \left(0.7705,0.0925\right)\\ \left(0.0513,0.9006\right)& \left(0.2124,0.7208\right)& \left(0.5000,0.5000\right)& \left(0.6210,0.2587\right)& \left(0.7109,0.1047\right)\\ \left(0.0679,0.8441\right)& \left(0.0860,0.8088\right)& \left(0.2587,0.6210\right)& \left(0.5000,0.5000\right)& \left(0.6001,0.2510\right)\\ \left(0.0760,0.8055\right)& \left(0.0925,0.7705\right)& \left(0.1047,0.7109\right)& \left(0.2510,0.6001\right)& \left(0.5000,0.5000\right)\end{array}\right]$

By calculating the distance between $\overline{R}$ and $R$, we obtain $d\left({\overline{R}}_A,R_A\right)=0.1702>0.1$, which indicates that $R_A$ does not pass the consistency test. Then, we need to perform steps 2.1–2.2. The number of iterations $p$ is initialized to be 1, $p=1$, and let $R^{\left(1\right)}=R$.

Step 2.1: Calculate

(3)$d\left(\overline{R},R^{\left(p\right)}\right)=\frac{1}{2\left(m-1\right)\left(m-2\right)}{\displaystyle \sum }_{i=1}^n{\displaystyle \sum }_{j=1}^n\left(\left|{\overline{u}}_{ij}-u_{ij}^{\left(p\right)}\right|+\left|{\overline{v}}_{ij}-v_{ij}^{\left(p\right)}\right|+\left|{\overline{\pi }}_{ij}-{\pi }_{ij}^{\left(p\right)}\right|\right)$

If $d\left(\overline{R},R^{\left(p\right)}\right)<0.1$, output $R^{\left(p\right)}$ and go to step 3; otherwise, go to step 2.2.

Step 2.2: Construct the synthetic intuitionistic fuzzy judgment matrix ${\tilde{R}}^{\left(P\right)}={\left({\tilde{r}}_{ij}^{\left(p\right)}\right)}_{n\times n}$ such that ${\tilde{r}}_{ij}^{\left(p\right)}=\left({\tilde{u}}_{ij}^{\left(p\right)},{\tilde{v}}_{ij}^{\left(p\right)}\right)$, and

(4)${\tilde{u}}_{ij}^{\left(p\right)}=\frac{{\left(u_{ij}^{\left(p\right)}\right)}^{1-\sigma }{\left({\overline{u}}_{ij}\right)}^{\sigma }}{{\left(u_{ij}^{\left(p\right)}\right)}^{1-\sigma }{\left({\overline{u}}_{ij}\right)}^{\sigma }+{\left(1-u_{ij}^{\left(p\right)}\right)}^{1-\sigma }{\left(1-{\overline{u}}_{ij}\right)}^{\sigma }}\hspace{1em}i,j=1,2,\cdots ,n,$

(5)${\tilde{v}}_{ij}^{\left(p\right)}=\frac{{\left(v_{ij}^{\left(p\right)}\right)}^{1-\sigma }{\left({\overline{v}}_{ij}\right)}^{\sigma }}{{\left(v_{ij}^{\left(p\right)}\right)}^{1-\sigma }{\left({\overline{v}}_{ij}\right)}^{\sigma }+{\left(1-v_{ij}^{\left(p\right)}\right)}^{1-\sigma }{\left(1-{\overline{v}}_{ij}\right)}^{\sigma }}\hspace{1em}i,j=1,2,\cdots ,n,$

Let $R^{\left(p+1\right)}={\tilde{R}}^{\left(P\right)}$, i.e., $u_{ij}^{\left(p+1\right)}={\tilde{u}}_{ij}^{\left(p\right)}$, $v_{ij}^{\left(p+1\right)}={\tilde{v}}_{ij}^{\left(p\right)}$, and let $p=p+1$; then, go to step 2.1.

In the above step, let $\sigma =0.8$; based on Equations (4) and (5), the final intuitive fuzzy judgment matrix:

$R_A{}^{\left(2\right)}={\tilde{R}}_A{}^{\left(1\right)}=\left[\begin{array}{ccccc}\left(0.5000,0.5000\right)& \left(0.7782,0.1670\right)& \left(0.8753,0.0679\right)& \left(0.8168,0.0884\right)& \left(0.7720, 0.0981\right)\\ \left(0.1670,0.7782\right)& \left(0.5000,0.5000\right)& \left(0.7208,0.2124\right)& \left(0.7848,0.1071\right)& \left(0.7479,0.1131\right)\\ \left(0.0679,0.8753\right)& \left(0.2124,0.7208\right)& \left(0.5000,0.5000\right)& \left(0.6210,0.2587\right)& \left(0.6884,0.1280\right)\\ \left(0.0884,0.8168\right)& \left(0.1071,0.7848\right)& \left(0.2587,0.6210\right)& \left(0.5000,0.5000\right)& \left(0.6001,0.2510\right)\\ \left(0.0981,0.7720\right)& \left(0.1131,0.7479\right)& \left(0.1280,0.6884\right)& \left(0.2510,0.6001\right)& \left(0.5000,0.5000\right)\end{array}\right]$

Now, we have $d\left({\overline{R}}_A,R_A{}^{\left(2\right)}\right)=0.0260<0.1$, so $R_A{}^{\left(2\right)}$ passes the consistency test. The same method is used to test the consistency of the intuitionistic fuzzy judgment matrix for the second-level indicators. The value of parameter $\sigma$ is changed to make it pass the test.

Step 3: Determine the weights of the first-level indicators. The two-dimensional weight ${\omega }_i$ of the second-level indicators are obtained using Equation (6):

(6)${\omega }_i=\left[\frac{{{\displaystyle \sum }}_{i=1}^n{u}_{ij}}{{{\displaystyle \sum }}_{i=1}^n{{\displaystyle \sum }}_{j=1}^n\left(1-v_{ij}\right)},1-\frac{{{\displaystyle \sum }}_{j=1}^n\left(1-v_{ij}\right)}{{{\displaystyle \sum }}_{i=1}^n{{\displaystyle \sum }}_{j=1}^n{u}_{ij}}\right]\hspace{1em}i=1,2,3,\cdots n$

Then, the two-dimensional indicator weights are fuzzily transformed from a vague set to a fuzzy set, and the two-dimensional weights are made one-dimensional by Equations (7) and (8).

(7)$G_i=\frac{u_i}{u_i+v_i}\hspace{1em}\hspace{1em}i=1,2,3,\cdots n$

(8)${\phi }_i=\frac{G_i}{{{\displaystyle \sum }}_{i=1}^n{G}_i}\hspace{1em}\hspace{1em}i=1,2,3,\cdots n$

After the above steps, we have the weights of the first-level indicators: ${\phi }_{A1}=0.332$, ${\phi }_{A2}=0.256, {\phi }_{A3}=0.182,{\phi }_{A4}=0.135, \mathrm{and} {\phi }_{A5}=0.095$. These weights correspond to the consumers’ shopping risk in the commodity dimension, the live-streamer dimension, the platform dimension, the payment dimension, and the logistics dimension, respectively. The weights of the second-level indicators can be obtained using the same method, and the results are shown in Table 1. Among the first-level indicators, the risk weight of the commodity dimension is the largest. The risks of the other four dimensions also have certain weights, which supports our hypothesis (1).

4.2. Quantifying Shopping Risks Based on Cloud Model

A cloud model is a cognitive model that enables a bidirectional conversion between qualitative concepts and quantitative instantiations [40]. We used the cloud model to quantify shopping risks. We let typical users rate the minimum and maximum consumers’ shopping risks for the four platforms with a value range of $\left[0, 100\right]$. The higher the score, the greater the risk. For ease of calculation, the rating data were normalized to $\left[0, 1\right]$. The reverse cloud generator was used to convert quantitative data into qualitative concepts, and we obtained digital cloud features $\left(Ex,En,He\right)$, where $Ex$, $En$, and $He$ are the expectation, entropy, and hyperentropy of risk values, respectively [41].

The final numerical characteristics of the cloud model for the four live streaming commerce platforms are shown in Table 2. The ranking of the comprehensive shopping risk based on the $Ex$ value is: Kuaishou > Douyin > Taobao > JD.com, which supports hypothesis (2).

Figure 3 shows several cloud droplets of the normal distribution cloud model that can be generated on the two-dimensional coordinate system through the forward cloud generator based on the three numerical characteristics of the cloud in Table 2. The horizontal axis represents the sample value of the concept of “consumers’ shopping risk evaluation value”, and the expected value is the point that best represents the concept. The vertical axis represents the degree to which the sample point belongs to the concept. The black cloud droplets represent the five levels of risk evaluation divided by the golden ratio. Level I represents “low risk” (0, 0.103, 0.0131), level II represents “relatively low risk” (0.309, 0.064, 0.0081), level III represents “general risk” (0.5, 0.0039, 0.005), level IV represents “relatively high risk” (0.691, 0.064, 0.0081), and level V represents “high risk” (1, 0.103, 0.0131). The shopping risks of consumers on four platforms are represented by cloud droplets of different colors, where green, red, blue, and yellow represent the risks of consumers shopping on Taobao, Douyin, Kuaishou, and JD.com, respectively. Intuitively, the different colored cloud droplets belong to the risk level corresponding to the closest black cloud droplet. The comprehensive risk evaluation cloud models of the four live-streaming commerce platforms are all distributed between “low risk” and “relatively low risk”. In Figure 3a, the $Ex$ values of JD.com and Taobao are 0.1248 and 0.0942, respectively, so their comprehensive risk levels are closer to “low risk”. The $Ex$ values of Douyin and Kuaishou are 0.2664 and 0.2828, respectively, so their comprehensive risk levels are closer to “relatively low risk”.

Figure 3b shows the consumers’ commodity dimension risk on the four platforms. The four platforms are listed in ascending order of risk by commodity dimension: JD.com, Taobao, Douyin, and Kuaishou. The $Ex$ of Taobao is 0.143, which is between “low risk” and “relatively low risk”. Although Taobao has a wide variety of commodities with higher shopping attributes, it also leads, to a certain extent, to an uneven commodity quality. The $Ex$ values of Douyin and Kuaishou are 0.2695 and 0.2697, which is relatively high because they are not strict enough in their merchant review and entry mechanisms, leading to poor quality control and confusing commodity prices. JD.com has the lowest $Ex$ of 0.0848 because they are willing to endorse the quality of their products and have a price-protection mechanism, making them the platform with the lowest product risk.

As shown in Figure 3c, the live-streamer dimension risk on the Taobao platform is the lowest among the four platforms, and its $Ex$ is 0.0911. However, the “Matthew effect” of Taobao live streamers can lead to an imbalance between the top and bottom live streamers on the platform. The identities of the live streamers on Kuaishou and Douyin are more diversified and civilianized, but their professional abilities are not strong, which leads to their $Ex$ being close to 0.3. In addition, although the $Ex$ of JD.com is 0.1054, which is already relatively low, its types of live streamers are too homogeneous, leaving a large gap in meeting user needs.

Figure 3d describes the platform dimension risk consumers face when shopping on live-streaming commerce platforms. The $Ex$ values of the platform’s own risk for the four platforms Taobao, Douyin, Kuaishou, and JD.com are 0.1406, 0.2879, 0.3077, and 0.0971, respectively. Among them, Kuaishou’s own risk is the highest with its $Ex$ of over 0.3, and Taobao’s own risk is the lowest. Consumers face relatively significant risks in terms of interface design and unreasonable functionality on these four platforms. For example, Taobao’s Share Live Room is a bit more complicated to operate. Douyin’s e-commerce system is weak. Kuaishou’s interface is not simple. The platform’s service feedback is bad. JD.com’s live buttons are scattered, and the live room entrances are few.

The payment dimension risk in Figure 3e shows that the payment dimension’s risk-prevention technologies of these four platforms are all up to standard, but there are still differences. Taobao’s payment system is relatively perfect because Alipay offers a shopping guarantee for it. Every transaction on JD.com will have real-time reminders with high security. In contrast, the $Ex$ values of Douyin and Kuaishou are 0.2487 and 0.2876, respectively. The payment types covered by Douyin and Kuaishou include scenarios such as rewards and mostly rely on third-party payment institutions, which makes it inconvenient for them to regulate their payment systems.

The logistics dimension risk is shown in Figure 3f. The logistics risk faced by consumers shopping on JD.com is close to being “low risk” because JD.com’s distribution mode combines its own logistics system with third-party logistics to provide high-quality distribution services. The Taobao, Douyin, and Kuaishou platforms cooperate with third-party logistics companies, resulting in none of their logistics risks being low enough to be close to “relatively low risk”. Among them, Taobao’s distribution cost is relatively high. Douyin has more end-of-line distribution problems.

In summary, in live-streaming commerce, consumers’ shopping risks on JD.com are very low in all dimensions. Taobao’s commodity dimension, platform dimension, and logistics dimension risks are relatively high. Douyin and Kuaishou’s risks in all dimensions are higher, but they are still within control.

5. Discussion

5.1. Theoretical Implications

Our study combined two new risk evaluation methods, i.e., an intuitionistic fuzzy hierarchical analysis [42,43] and cloud modeling [41]. For the first time, these methods were applied to study the risks of live-streaming commerce, an emerging e-commerce model. Our proposed multidimensional evaluation method of consumer shopping risk provides a set of tools for the live-streaming commerce platform that can effectively quantify and evaluate the consumer’s shopping risks on the platform.

Like other risk evaluation studies [4,43], we constructed a multilayered consumer shopping risk evaluation index system containing five first-level indicators and eighteen second-level indicators. In addition to the conventional platform’s own dimensions, we also considered the logistics and payment dimensions mentioned in [6] and the commodity dimension mentioned in [7,8]. Especially because we studied live e-commerce, we also considered the live-streamer dimension. Our index system comprehensively reflects the different stakeholders and risk factors involved in the live-streaming commerce mode and provides a scientific basis for consumer shopping risk evaluation.

Our research fills the gap in risk evaluation in the field of live-streaming commerce and enriches the methods of e-commerce risk evaluation. Our study is of great significance for promoting the sustainable development of live-streaming commerce and protecting consumer rights.

5.2. Managerial Implications

Our proposed framework can assist merchants, live streamers, live-streaming commerce platforms, and relevant regulatory bodies in adopting targeted measures to promote the healthy and orderly development of the live-streaming commerce industry. The results of this paper remind platforms and consumers of the following aspects of live streaming commerce:

(1) Commodity dimension. The platforms should strengthen their review of merchants, prevent unqualified products from entering the platform, and ensure the quality of products and services to reduce the risk of online shopping for consumers. Consumers should choose regular live-streaming commerce platforms and merchant stores to purchase products with a complete understanding of product information.

(2) Live-streamer dimension. The platforms should strengthen their review and management of live streamers, prohibit live streamers from exaggerating product promotion and falsifying data traffic, etc. They also need to provide consumers with real product and service information. Consumers should abandon their blind trust in individual live streamers and avoid following trends when shopping. For live streamers, it is crucial to enhance their professional competence and cultivate a sense of trust with consumers.

(3) Platform dimension. The platforms should improve the friendliness of their interfaces, optimize the consumer service experience, and comply with relevant privacy protection policies. Through screenshots and screen recordings, consumers can retain shopping-related evidence to facilitate the after-sales maintenance of their legitimate rights and interests.

(4) Payment dimension. The platforms should improve their payment systems, do an excellent job managing transaction security, and implement payment risk prevention measures to improve the security of platform transactions. Consumers should pay attention to the safety of personal information and property when shopping online and consciously resist related illegal and criminal behaviors.

(5) Logistics dimension. The platforms should improve their logistics systems and increase the speed of logistics delivery to ensure the integrity of products and the cost-effectiveness of logistics costs. Consumers should check products in time after receiving them, give feedback, and deal with problems in time. They should use legal weapons to protect their legitimate rights and interests when necessary.

Furthermore, this paper quantitatively evaluated consumers’ shopping risks on four typical live-streaming commerce platforms—Taobao, Douyin, Kuaishou, and JD.com—for the first time. We revealed the risk levels of these platforms under different dimensions and indicators, providing a reference for consumers to choose suitable platforms and products, as well as for these platforms to improve service quality and reduce risks.

6. Conclusions and Future Research

In order to quantitatively evaluate consumers’ shopping risk on live streaming commerce platforms, an analysis framework based on an IFAHP–cloud model was proposed. In our framework, we constructed a multidimensional consumer shopping risk evaluation index system that considered different stakeholders involved in live-streaming commerce. The index system consisted of five first-level and eighteen second-level indicators. The IFAHP–cloud model was used to quantify consumers’ shopping risks on four typical live-streaming commerce platforms in China.

Our results revealed that in the live-streaming commerce risk evaluation indicators, the commodity dimension had the highest risk weight, while the logistics dimension had the lowest. The comprehensive risks of each platform were: Kuaishou > Douyin > Taobao > JD.com. The analysis results provided targeted recommendations for merchants, live-streaming platforms, live-commerce platforms, and relevant management departments and helped promote the sustainable development of the e-commerce industry.

Live-streaming commerce, as a new e-commerce mode, is still in the rapid development stage. Risk factors outside the assessment framework of this study may be emerging. Therefore, our framework is slightly less flexible. In addition, although our framework was applied to four representative live-streaming commerce platforms, more platforms still need to be evaluated to generate more generalized results.

In the future, more types of data, such as product comments and user behavior data generated when using live streaming commerce platforms, can be collected to make the risk evaluation more effective. Future research will also consider more live-streaming commerce platforms. It will also be interesting to explore the impact of different risk factors on consumer decision-making.

Author Contributions

Conceptualization, H.L.; methodology, H.L. and Z.W.; validation, H.L. and Z.W.; writing—original draft preparation, H.L. and Z.W.; writing—review and editing, H.L., Z.W., Z.Y. and X.Y.; visualization, H.L. and Z.W.; supervision, H.L.; project administration, H.L.; funding acquisition, H.L. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy.

Acknowledgments

The authors thank the editor and reviewers for providing valuable suggestions that have improved the quality of this paper.

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Analysis framework for evaluating consumers’ shopping risks in live-streaming commerce. Source: the authors.

Figure 2. Consumers’ shopping risk evaluation index system in live-streaming commerce. Source: the authors.

Figure 3. Cloud diagram of consumers’ shopping risks.

Table 1

Weights of consumers’ shopping risk evaluation index in live-streaming commerce.

First-Level Indicator	Weight	Second-Level Indicator	Weight
Commodity dimension risk (A1)	0.332	Counterfeiting and inferior (B1)	0.391
		Unreasonable price (B2)	0.275
		No goods received (B3)	0.196
		Refusing to be returned or exchanged (B4)	0.138
Live-streamer dimension risk (A2)	0.256	Poor professional ability (B5)	0.486
		Lack of credit (B6)	0.310
		Shirking responsibility after sales (B7)	0.204
Platform dimension risk (A3)	0.182	Unfriendly interface design (B8)	0.387
		Unreasonable function settings (B9)	0.258
		Poor service feedback (B10)	0.199
		Weak personal privacy protection (B11)	0.156
Payment dimension risk (A4)	0.135	Private payment method (B12)	0.388
		Financial loss caused by payment (B13)	0.147
		Payment interruption or cancellation (B14)	0.269
		Payment data tampering or theft (B15)	0.196
Logistics dimension risk (A5)	0.095	Slow delivery speed (B16)	0.481
		Damage and loss of goods (B17)	0.314
		High logistics service costs (B18)	0.205

Table 2

The numerical characteristics of the cloud model for four live-streaming commerce platforms.

Risk Dimension	Platform	$\mathit{E}\mathit{x}$	$\mathit{E}\mathit{n}$	$\mathit{H}\mathit{e}$
Comprehensive	Taobao	0.1248	0.0563	0.0065
	Douyin	0.2664	0.0543	0.0087
	Kuaishou	0.2828	0.0516	0.0103
	JD.com	0.0942	0.0527	0.0062
Commodity	Taobao	0.1430	0.0598	0.0074
	Douyin	0.2695	0.0529	0.0088
	Kuaishou	0.2697	0.0493	0.0116
	JD.com	0.0848	0.0550	0.0058
Live streamer	Taobao	0.0911	0.0525	0.0053
	Douyin	0.2431	0.0585	0.0100
	Kuaishou	0.2688	0.0523	0.0114
	JD.com	0.1054	0.0482	0.0040
Platform	Taobao	0.1406	0.0565	0.0061
	Douyin	0.2879	0.0507	0.0055
	Kuaishou	0.3077	0.0562	0.0101
	JD.com	0.0971	0.0541	0.0096
Payment	Taobao	0.0914	0.0532	0.0092
	Douyin	0.2487	0.0531	0.0076
	Kuaishou	0.2876	0.0549	0.0064
	JD.com	0.0927	0.0521	0.0101
Logistics	Taobao	0.1689	0.0480	0.0085
	Douyin	0.3023	0.0563	0.0101
	Kuaishou	0.3119	0.0508	0.0060
	JD.com	0.0938	0.0534	0.0085

References

1. CNNIC (China Internet Network Information Center). Available online: https://cnnic.cn/n4/2023/0303/c88-10757.html (accessed on 6 June 2023).

2. TechNode Feed. Available online: https://technode.com/2022/10/25/chinese-influencer-li-jiaqi-leads-singles-day-livestream-pre-sales/ (accessed on 6 June 2023).

3. Karoui, K. Security novel risk assessment framework based on reversible metrics: A case study of DDoS attacks on an E-commerce web server. Int. J. Netw. Manag.; 2016; 26, pp. 553-578. [DOI: https://dx.doi.org/10.1002/nem.1956]

4. Zhang, Y.; Deng, X.; Wei, D.; Deng, Y. Assessment of E-Commerce security using AHP and evidential reasoning. Expert Syst. Appl.; 2012; 39, pp. 3611-3623. [DOI: https://dx.doi.org/10.1016/j.eswa.2011.09.051]

5. Dinev, T.; Hart, P. An extended privacy calculus model for e-commerce transactions. Inf. Syst. Res.; 2006; 17, pp. 61-80. [DOI: https://dx.doi.org/10.1287/isre.1060.0080]

6. Song, B.; Yan, W.; Zhang, T. Cross-border e-commerce commodity risk assessment using text mining and fuzzy rule-based reasoning. Adv. Eng. Inform.; 2019; 40, pp. 69-80. [DOI: https://dx.doi.org/10.1016/j.aei.2019.03.002]

7. Liu, Y.; Lu, J.; Mao, F.; Tong, K. The product quality risk assessment of e-commerce by machine learning algorithm on spark in big data environment. J. Intell. Fuzzy Syst.; 2019; 37, pp. 4705-4715. [DOI: https://dx.doi.org/10.3233/JIFS-179305]

8. Mou, J.; Cui, Y.; Kurcz, K. Trust, risk and alternative website quality in B-buyer acceptance of cross-border E-commerce. J. Glob. Inf. Manag.; 2020; 28, pp. 167-188. [DOI: https://dx.doi.org/10.4018/JGIM.2020010109]

9. Anastasiei, B.; Dospinescu, N.; Dospinescu, O. Word-of-Mouth Engagement in Online Social Networks: Influence of Network Centrality and Density. Electronics; 2023; 12, 2857. [DOI: https://dx.doi.org/10.3390/electronics12132857]

10. Al Naim, A.F.; Sobaih, A.E.E.; Elshaer, I.A. Enhancing Green Electronic Word-of-Mouth in the Saudi Tourism Industry: An Integration of the Ability, Motivation, and Opportunity and Planned Behaviour Theories. Sustainability; 2023; 15, 9085. [DOI: https://dx.doi.org/10.3390/su15119085]

11. Khasawneh, M.H.A.; Rabata, A. The impact of augmented reality on behavioural intention and E-WOM. Int. J. Electron. Bus.; 2023; 18, pp. 194-225. [DOI: https://dx.doi.org/10.1504/IJEB.2023.130165]

12. Cheng, G.; Cherian, J.; Sial, M.S.; Mentel, G.; Wan, P.; Álvarez-Otero, S.; Saleem, U. The relationship between csr communication on social media, purchase intention, and e-wom in the banking sector of an emerging economy. J. Theor. Appl. Electron. Commer. Res.; 2021; 16, pp. 1025-1041. [DOI: https://dx.doi.org/10.3390/jtaer16040058]

13. Dong, X.; Zhao, H.; Li, T. The role of live-streaming e-commerce on consumers’ purchasing intention regarding green agricultural products. Sustainability; 2022; 14, 4374. [DOI: https://dx.doi.org/10.3390/su14074374]

14. Lu, B.; Chen, Z. Live streaming commerce and consumers’ purchase intention: An uncertainty reduction perspective. Inf. Manag.; 2021; 58, 103509. [DOI: https://dx.doi.org/10.1016/j.im.2021.103509]

15. Sun, Y.; Shao, X.; Li, X.; Guo, Y.; Nie, K. How live streaming influences purchase intentions in social commerce: An IT affordance perspective. Electron. Commer. Res. Appl.; 2019; 37, 100886. [DOI: https://dx.doi.org/10.1016/j.elerap.2019.100886]

16. Ma, X.; Zou, X.; Lv, J. Why do consumers hesitate to purchase in live streaming? A perspective of interaction between participants. Electron. Commer. Res. Appl.; 2022; 55, 101193. [DOI: https://dx.doi.org/10.1016/j.elerap.2022.101193]

17. He, Y.; Li, W.; Xue, J. What and how driving consumer engagement and purchase intention in officer live streaming? A two-factor theory perspective. Electron. Commer. Res. Appl.; 2022; 56, 101223. [DOI: https://dx.doi.org/10.1016/j.elerap.2022.101223]

18. Wang, D.; Luo, X.R.; Hua, Y.; Benitez, J. Customers’ help-seeking propensity and decisions in brands’ self-built live streaming E-Commerce: A mixed-methods and fsQCA investigation from a dual-process perspective. J. Bus. Res.; 2023; 156, 113540. [DOI: https://dx.doi.org/10.1016/j.jbusres.2022.113540]

19. Gong, H.; Zhao, M.; Ren, J.; Hao, Z. Live streaming strategy under multi-channel sales of the online retailer. Electron. Commer. Res. Appl.; 2022; 55, 101184. [DOI: https://dx.doi.org/10.1016/j.elerap.2022.101184]

20. Wongkitrungrueng, A.; Assarut, N. The role of live streaming in building consumer trust and engagement with social commerce sellers. J. Bus. Res.; 2020; 117, pp. 543-556. [DOI: https://dx.doi.org/10.1016/j.jbusres.2018.08.032]

21. Zhang, T.; Tang, Z.; Han, Z. Optimal online channel structure for multinational firms considering live streaming shopping. Electron. Commer. Res. Appl.; 2022; 56, 101198. [DOI: https://dx.doi.org/10.1016/j.elerap.2022.101198]

22. Naiyi, Y.E. Dimensions of consumer’s perceived risk in online shopping. J. Electron. Sci. Technol.; 2004; 2, pp. 177-182.

23. Lin, M.J.; Wang, W.T. Examining e-commerce customer satisfaction and loyalty: An integrated quality-risk-value perspective. J. Organ. Comput. Electron. Commer.; 2015; 25, pp. 379-401. [DOI: https://dx.doi.org/10.1080/10919392.2015.1089681]

24. Pappas, N. Marketing strategies, perceived risks, and consumer trust in online buying behaviour. J. Retail. Consum. Serv.; 2016; 29, pp. 92-103. [DOI: https://dx.doi.org/10.1016/j.jretconser.2015.11.007]

25. Peng, H. Construction basis of C2C e-commerce credit evaluation index. J. Electron. Commer. Organ.; 2017; 15, pp. 11-23. [DOI: https://dx.doi.org/10.4018/JECO.2017100102]

26. Ramanathan, R. The moderating roles of risk and efficiency on the relationship between logistics performance and customer loyalty in e-commerce. Transp. Res. Part E Logist. Transp. Rev.; 2010; 46, pp. 950-962. [DOI: https://dx.doi.org/10.1016/j.tre.2010.02.002]

27. Ramanathan, R. An empirical analysis on the influence of risk on relationships between handling of product returns and customer loyalty in E-commerce. Int. J. Prod. Econ.; 2011; 130, pp. 255-261. [DOI: https://dx.doi.org/10.1016/j.ijpe.2011.01.005]

28. Dimoka, A.; Hong, Y.; Pavlou, P.A. On product uncertainty in online markets: Theory and evidence. MIS Q.; 2012; 36, pp. 395-426. [DOI: https://dx.doi.org/10.2307/41703461]

29. Xing, Q.; Ren, T.; Deng, F. Analysis of the transaction behavior of live broadcasters with goods based on the multi-stage game under dynamic credit index. Sustainability; 2023; 15, 4233. [DOI: https://dx.doi.org/10.3390/su15054233]

30. Hasan, L.; Morris, A.; Probets, S. A comparison of usability evaluation methods for evaluating e-commerce websites. Behav. Inf. Technol.; 2012; 31, pp. 707-737. [DOI: https://dx.doi.org/10.1080/0144929X.2011.596996]

31. Venkatesh, V.; Agarwal, R. Turning visitors into customers: A usability-centric perspective on purchase behavior in electronic channels. Manag. Sci.; 2006; 52, pp. 367-382. [DOI: https://dx.doi.org/10.1287/mnsc.1050.0442]

32. Hausman, A.V.; Siekpe, J.S. The effect of web interface features on consumer online purchase intentions. J. Bus. Res.; 2009; 62, pp. 5-13. [DOI: https://dx.doi.org/10.1016/j.jbusres.2008.01.018]

33. Ong, C.E.; Teh, D. Redress procedures expected by consumers during a business-to-consumer e-commerce dispute. Electron. Commer. Res. Appl.; 2016; 17, pp. 150-160. [DOI: https://dx.doi.org/10.1016/j.elerap.2016.04.006]

34. See-To, E.W.; Ho, K.K. A study on the impact of design attributes on E-payment service utility. Inf. Manag.; 2016; 53, pp. 668-681. [DOI: https://dx.doi.org/10.1016/j.im.2016.02.004]

35. Kim, C.; Tao, W.; Shin, N.; Kim, K.S. An empirical study of customers’ perceptions of security and trust in e-payment systems. Electron. Commer. Res. Appl.; 2010; 9, pp. 84-95. [DOI: https://dx.doi.org/10.1016/j.elerap.2009.04.014]

36. Cui, R.; Li, M.; Li, Q. Value of high-quality logistics: Evidence from a clash between SF Express and Alibaba. Manag. Sci.; 2020; 66, pp. 3879-3902. [DOI: https://dx.doi.org/10.1287/mnsc.2019.3411]

37. Xiao, T.; Qi, X. A two-stage supply chain with demand sensitive to price, delivery time, and reliability of delivery. Ann. Oper. Res.; 2016; 241, pp. 475-496. [DOI: https://dx.doi.org/10.1007/s10479-012-1085-6]

38. CCA (China Consumers Association). Available online: http://www.cca.org.cn/jmxf/detail/29533.html (accessed on 6 June 2023).

39. Xu, Z.; Liao, H. Intuitionistic fuzzy analytic hierarchy process. IEEE Trans. Fuzzy Syst.; 2013; 22, pp. 749-761. [DOI: https://dx.doi.org/10.1109/TFUZZ.2013.2272585]

40. Li, D.; Liu, C.; Gan, W. A new cognitive model: Cloud model. Int. J. Intell. Syst.; 2009; 24, pp. 357-375. [DOI: https://dx.doi.org/10.1002/int.20340]

41. Liu, H.C.; Wang, L.E.; Li, Z.; Hu, Y.P. Improving risk evaluation in FMEA with cloud model and hierarchical TOPSIS method. IEEE Trans. Fuzzy Syst.; 2018; 27, pp. 84-95. [DOI: https://dx.doi.org/10.1109/TFUZZ.2018.2861719]

42. Tumsekcali, E.; Ayyildiz, E.; Taskin, A. Interval valued intuitionistic fuzzy AHP-WASPAS based public transportation service quality evaluation by a new extension of SERVQUAL Model: P-SERVQUAL 4.0. Expert Syst. Appl.; 2021; 186, 115757. [DOI: https://dx.doi.org/10.1016/j.eswa.2021.115757]

43. Yu, X.; Zheng, D.; Zhou, L. Credit risk analysis of electricity retailers based on cloud model and intuitionistic fuzzy analytic hierarchy process. Int. J. Energy Res.; 2021; 45, pp. 4285-4302. [DOI: https://dx.doi.org/10.1002/er.6090]