Using Multi-Criteria Decision Making in Quality Function Deployment for Offshore Renewable Energies

Abstract

Quality function deployment (QFD) is now used in various fields, such as product development, design, manufacturing, planning, and quality management services, as a planning tool to achieve customer requirements and needs while improving performance and sustainability concerns. This paper presents a systematic literature review of multi-criteria decision-making (MCDM) methodologies integrated into QFD over the last year. In 2022, 56 research papers on planning strategies, the supply chain, and product development using QFD were published. Other fields such as energy, academia, and environment have also shown considerable interest in the integration of MCDM methodologies in QFD to improve decision-making processes. This research shows that the analytic hierarchy process (AHP) and the technique for order preference by similarity to ideal solutions (TOPSIS) methodologies are mainly used to rank customer requirements and weigh their importance in the house of quality (HoQ) structure. The use of fuzzy logic has been incorporated into the correlation matrix to evaluate the relationships between customer requirements and technical requirements. Methodologies such as decision-making trial and evaluation laboratory (DEMATEL) and fuzzy cognitive maps are implemented to deal with contradictions, and they have also been used to rank engineering characteristics. In the field of energy and renewable technologies, only few studies related to the integration of MCDM methodologies in QFD were found, but it is forecasted that their use will be used more often as they offer improvements and benefits in the ocean energy sector.

1. Introduction

Quality function deployment (QFD) is a planning tool for the design, development, and implementation of new products that takes into consideration customer needs and values [1]. The methodology was developed by Yoji Akao and Shigeru Mizuno in the 1960s at the Tokyo Institute of Technology. In 1972, the QFD methodology was used in the design of oil tanks for Mitsubishi Heavy Industries, allowing the company to reduce costs.

QFD methodology was initially used to develop high-quality products to meet customer requirements and needs. However, the methodology was expanded to different fields, such as design, planning, decision making, engineering management, quality management, teamwork, timing, and costing. Furthermore, QFD as a planning tool has been applied to transportation and communication, electronics and electrical utilities, manufacturing, software systems, services, and education and research [2]. Quantitative and multi-criteria decision-making (MCDM) methods have been integrated in the QFD methodology to improve objectiveness to define customer importance, prioritize customer needs, and rate customer requirements (WHATs) and technical requirements (HOWs) in the HoQ structure. The term multi-criteria decision making (MCDM) describes the process of selecting the best alternative among many, based on several criteria or attributes, that may be tangible or subjective [3]. Some of these methods include the analytic hierarchy process (AHP) and fuzzy logic methods [2].

To achieve the Sustainable Development Goals (SDGs) 7, 11, 13, and 14 from the 2030 agenda and net zero emissions by 2050, the transition to renewable electricity production must be addressed. In the study carried out by [4], the technical potential of offshore wind was estimated at 55,000 TWh/yr, wave potential 1000 TWh/yr, and tidal stream potential 180 TWh/yr. Advances in materials to manufacture devices such as tidal turbines have been challenging due to the technical requirements; however the efforts to improve the existing devices have the attention of stakeholders and engineers.

A recent case study in India [5] identified fourteen barriers to developing ocean energy, detailing them in the broad categories of technological, environmental, financial, infrastructural, socio-economic, political, and administrative. To overcome these obstacles, holistic frameworks concerning the design and manufacturing of ORE technologies are needed. From [5], the main technical requirements and targets in any ocean energy technology project can be established.

This paper is structured as follows: Section 2 presents the existing literature related to MCDM methodologies in quality function deployment and Section 3 describes the methodology applied in this systematic literature review by using PRISMA 2020 as a guideline. Section 4 presents the results, discussion, and findings of this research. Furthermore, a proposed framework for ORE technologies is presented in the final section.

2. Background

The recent literature review of [6] focuses on the integration of multi-criteria decision-making approaches into QFD methodology. The authors examined the period 2004 to 2021 and selected three data sources: Web of Science, Scopus, and Google Scholar, assessing a total of 997 papers from which a set of 59 articles were considered in the literature review. Their findings indicate that MCDM methodologies are integrated in QFD notably in fields such as manufacturing, supply chains, healthcare, education, and financial services that include hybrid methodologies [6].

A bibliometric and literature review of QFD was also carried out by [7] to assess articles from 1999 to 2020. Their screening process produced 396 articles for the literature analysis, and their results show that, in the last two decades, there was increasing interest in QFD, with the greatest number of publications in 2019: 45 articles.

The previous work has focused on the use of QFD methodologies for a number of industries; however, as a result of climate change and the imminent need for a decarbonized world, the present literature review seeks to identify methodologies that could be used in the early stages of a project for the growing installation of the offshore renewable energy (ORE) sector as decision-making tools integrated in QFD.

One of the most important aspects of QFD studies is the determination of the weightings in customer requirements (CRs). Often, the definition of the CR weights is made subjectively, and the relationship between the CR and engineering characteristics (ECs) is usually incomplete and vague. To improve QFD methodology, topics such as fuzzy set extensions, neutrosophic MCDM, the Kano model, and the Function Analysis System Technique (FAST) are used [8]. The main MCDM methodologies used in the current literature are summarized in the following sections.

2.1. The AHP Methodology

The analytic hierarchy process approach was developed by Thomas L. Saaty in the 1970s to deal with both the rational and instinctive processes used to select the best option among several alternatives, based on several criteria. The hierarchical structure of this method is the following: the goal is the top level, criteria the second, and the alternatives are to be evaluated thirdly. The AHP theory provides ratio scales from pairwise comparisons, with the fundamental 1–9 scale, which has been validated through theoretical justification in comparison of homogeneous elements [9]. Using pairwise comparisons, it is possible to judge which of two entities from a hierarchical structure is preferred. The AHP has been implemented in case studies in sectors such as transport, government, marketing, planning, product development, and economy, as well as to address social and political matters [10].

2.2. The ELECTRE Methodology

The elimination and choice translating reality is a multi-criteria decision analysis (MCDA) based on the concept of dealing with “outranking relations” by pairwise comparisons. The ELECTRE methodology uses a system of binary outranking relations between the alternatives presented. The first ELECTRE study was developed in 1996 by Benayoun and Sussman, with subsequent versions later, such as: ELECTRE I, ELECTRE II, ELECTRE III, ELECTRE IV, ELECTRE TRI, ELECTRE IS, and ELECTRE TRI-B [11].

The steps of the ELECTRE methodology are:

• Normalization of the decision matrix.

• Weighting the normalized decision matrix.

• Determination of the concordance and discordance sets.

• Construction of the concordance and discordance matrices.

• Determination of the concordance and discordance dominance matrices.

• Determination of the aggregate dominance matrix.

• Elimination of the less favorable alternatives [1].

2.3. The TOPSIS Methodology

The technique for order preference by similarity to ideal solutions (TOPSIS) method was developed by Hwang and Yoon in 1981, and is the second most used in decision-making processes after AHP [12]. TOPSIS is a multiple attribute decision-making (MADM) method because of its mathematical foundation and ease of application in a large number of fields, such as manufacturing, sustainability assessment, material selection, financial performance analysis, technology assessment, product selection, and machine selection. There are two approaches: TOPSIS and modified TOPSIS. The first is based on the premise that the best solution must be that nearest to the positive-ideal solution, and the negative-ideal solution is furthest from this. It is stated in [13] that modified TOPSIS is suitable for cases where reliable subjective weights cannot be obtained.

The stages of the TOPSIS method can be summarized as follows:

• Calculate the normalized performance ratings.

• Integrate weight with ratings.

• Identify positive and negative ideal solutions.

• Obtain the separation values of the distance from both positive and negative solutions.

• Calculate the overall preference score by ranking the alternatives.

The modified TOPSIS methodology uses the entropy-based weight elicitation method to obtain objective weights, and has four stages:

• Normalize the original decision matrix.

• Identify the ideal solutions—both the positive and negative.

• Obtain the weighted Euclidean distance.

• Obtain the overall performance score [14].

2.4. The VIKOR Methodology

The VIsekriterijumsko KOmpromisno Rangiranje (VIKOR) methodology was established by Opricovic in 1998 as an MCDM method used to solve a discrete multicriteria problem and to rank and select the best solution considering the existence of conflicting criteria conditions [15]. This technique allows both the alternatives closest and furthest from the ideal solution to be taken into account [16]. The implementation of fuzzy numbers into VIKOR improves the approach to achieve the main objective by reducing uncertainty.

The VIKOR method consist of the following steps:

• Determination of the f* (the best) and f- (the worst) indexes.

• Obtaining the S and R indexes for each alternative.

• Calculation of VIKOR index for each alternative.

• Ranking of alternatives [17].

2.5. The DEMATEL Methodology

The aim of the DEMATEL technique is to identify and map the interrelations of a set of options by separating the factors into two groups: those of cause and effect [18]. This method was developed by Fonetla and Gabius in 1971 to address complex issues. DEMATEL is also used to analyze the internal relations among the alternatives and it allows the qualitative attributes to be converted into quantitative attributes [19], using the following steps:

• Find the average matrix.

• Calculate the normalized initial direct relation matrix.

• Compute the total relation matrix.

• Obtain the threshold value and the interrelationship map [20].

In the systematic review carried out by Si et al. [21], several DEMATEL approaches were identified: classic DEMATEL, fuzzy DEMATEL, grey DEMATEL, ANP-DEMATEL, and others. One of the most commonly used approaches is that of the analytic network process (ANP) combined with DEMATEL. The authors reviewed 346 articles from 2006 to 2016, of which 44.5% of them described articles using this combination of approaches. With 30.3% of the articles they reviewed, the classic DEATEL method was the second most used approach, and fuzzy DEMATEL was third most used with 18.2%.

Other methodologies such as AHP, VIKOR, and TOPSIS were also combined with DEMATEL to address issues of fields including supply chains, planning, improvement of products and services, environment, and social and political issues.

2.6. The PROMETHEE Methodology

The preference ranking organization method for enrichment evaluations is an MCDA methodology developed by Jean-Pierre Brans in 1982, and later extended in 1985. It is used as an outranking tool for an established set of alternatives to select the best option according to defined criteria [22] and thus improve decision-making evaluation.

Variations of this approach are PROMETHEE I, PROMETHEE II, and PROMETHEE III. The first provides an examination concept to rank the alternatives; this is only partially developed (with a reduced number of steps to complete the process). On the other hand, in PROMETHEE II and PROMETHEE III, there is an extended approach used to obtain the final ranking of the alternatives. In all three cases, there is no need for independence of the attributes and qualitative attributes can be converted into quantitative attributes [23].

The steps of the PROMETHEE methodology are:

• Determination of deviation based on pairwise comparisons.

• Application of the preference function.

• Calculation of the global preference index.

• Calculation of outranking flows.

• Calculation of net outranking flow.

In the PROMETHEE approach, the methodology includes steps 1–4, and, for PROMETHEE II and PROMETHEE III, steps 1–5 [22].

2.7. The ANP Methodology

The analytic network process approach was developed by Thomas L. Saaty in 1996 to establish a hierarchy of complex and interdependent elements. It has been applied to analyze risk and uncertainty in fields such as design, engineering and manufacturing systems, health, safety and environmental management, logistics, and supply chains [24]. It has a similar hierarchical structure to AHP, but considers all types of dependencies, and problems of interdependence, of the elements by constructing a network of criteria, sub-criteria, and other alternatives [25]. The main difference between AHP and ANP is that the first has a linear top-down structure with no feedback, while ANP receives feedback from clusters of the elements [26].

The following steps are used:

• Construction of the priority vectors.

• Construction of the super matrix.

• Construction of the cluster matrix.

• Obtainment of the weighted super matrix.

• Calculation of the limit super matrix.

• Calculation of the utility index.

• Determination of the final ranking of the alternatives [27].

Table 1 summarizes the main MCDM methodologies and their most common uses.

3. Systematic Review Methodology

This systematic review aims to address the following questions:

• Which are the most frequently used multicriteria decision-making approaches in quality function deployment methodology today?

• In what part of the house of quality structure of QFD are these MCDM approaches being used?

• In which fields are these approaches most commonly used in QFD?

• What is the most significant advantage of using MCDM approaches in QFD?

Figure 1 shows the stages of this review in a flow chart. The search query was first defined to include all keywords. Secondly, the database for the search stage was selected considering the state-of-the-art of the recently published systematic reviews. Then, the topics to be searched according to the string format TOPIC-PUBLICATION DATE-ABSTRACT were specified. The search string was chosen to give the greatest number of publications and the obtained data were stored and analyzed using EndNote and Excel. Finally, the results from the analysis are presented using tables and charts.

The MCDM methodologies considered in the search query are presented in Figure 2. They selected the search query for MCDM most frequently used in QFD. There are, of course, other MCDM approaches, such as fuzzy full consistency method (F-FUCOM) [35], double hierarchy hesitant linguistic term sets (DHHLTSs) [36], fuzzy Delphi method (FDM) [37,38,39,40], grey relation analysis (GRA) [6,37,41,42], and S-CoCoSo [43], that have been used in QFD studies.

This paper focuses on AHP, ELECTRE, TOPSIS, VIKOR, DEMATEL, PROMETHEE, and ANP because they are the methodologies that have been most widely applied in decision-making tools in the fields of product development, manufacturing, and sustainability [6].

3.1. The PRISMA Methodology

In this systematic review, the methodology PRISMA 2020 was used as a guideline to report systematic literature reviews and meta-analyses. Systematic reviews provide a synthesis of the state-of-the-art in many fields and can identify research gaps that could be filled in future studies [44]. From the 152 articles identified, 53 duplicate records and 1 retracted paper were removed. Then, in the stage of Screening, 94 of 98 articles were retrieved and then 21 were removed due to their lack of relevance to QFD approaches. The final set of 73 articles was composed of 69 journal articles and 4 conference proceedings. This methodology is summarized in Figure 3.

3.2. Definition of Keywords

During the search query, different combinations of methodologies were considered to identify the greatest number of articles; the keywords used during the search can be seen in Table 2.

4. Results and Discussion

This section gives an up-to-date report of the use of MCDM approaches in improving the HoQ of QFD methodology in 2022.

4.1. Publication Sources

Table 3 lists the 52 journals where 65 of the 69 articles identified were published. The remaining 4 articles were published as conference papers in Industrial Engineering and Operations Management, XXVIII IJCIEOM [45], Advances in Intelligent Systems and Computing [46,47], and Design, User Experience, and Usability: Design Thinking and Practice in Contemporary and Emerging Technologies, [48] and Usability proceedings in 2022.

The journal Sustainability from MDPI published five articles, followed by Mathematics with four articles and Computers & Industrial Engineering, IEEE Transactions on Engineering Management, and Journal of Intelligent & Fuzzy Systems with three publications each. The results illustrate the recent interest in the combination of MCDM with QFD in studies.

It is an area that has had a substantial impact on new mathematical methods and the implementation of specialized software to achieve QFD goals with MCDM methodologies.

4.2. MCDM Methodologies in QFD Identified

Figure 4 shows the MCDM methodologies most frequently used in QFD. AHP methodology is seen as the most frequently used in QFD to weigh the customer importance through a pairwise comparison [8,37,49,50,51,52,53,54,55,56,57]. Fuzzy logic theory has been used in QFD to rank customer requirements in many recent case studies [58,59,60,61,62]. TOPSIS comes second, being applied to rank customer requirements through the use of triangular fuzzy numbers [63,64], and as a robust approach integrated in other methodologies to improve the HoQ structure [58,59,65,66,67,68,69,70]. The MCDM approach that follows the number of applications reported is DEMATEL, and it is mainly used to deal with contradictions between technical requirements HOWs [8,37,46,71,72], but also to rank customer requirements [68,70]. Using DEMATEL to compute the importance of the customer requirements, the interrelationships between these elements can be analyzed simultaneously.

The study carried out by [70] demonstrated that DEMATEL can be useful in clarification and reducing uncertainties of the experts. Other methodologies have been incorporated into QFD, but their advancements are still in progress.

4.3. Other Methodologies

Newer methodologies have also been used in QFD in recent case studies, including hesitant fuzzy linguistics [36,73,74], fuzzy DEMATEL [39,75], grey relational analysis (GRA) [37,42,46,47,76], interval-valued fuzzy numbers (IVFN) [65,77], fuzzy Delphi method (FDM) [37,39,40,46,47,49,75], complex proportion assessment method (COPRAS) [78], ordinal priority approach-combinative distance-based assessment (OPA-CODAS) [79], social network analysis (SNA) [80], multi-stepwise weight assessment ratio analysis (M-SWARA) [81], entropy weight method (EWM) [38], double hierarchy hesitant linguistic term sets (DHHLTSs) [36], organísation, rangement et synthèse de données relarionnelles (ORESTE) [36], best-worst method-based [39,82], combined compromise solution (CoCoSo) [43,70], Dempster–Shafer Theory (DST) [83], fuzzy cognitive maps (FCM) [84], proportional hesitant fuzzy linguistic term sets (PHFLTSs) [85], cumulative prospect theory (CPT) [85], fuzzy measuring attractiveness by a categorically based evaluation technique (FMACBETH) [40], and fuzzy full consistency method (FUCOM) [35,86].

Figure 5 presents the percentage of hybrids and non-hybrids methodologies implemented in QFD. The use of more than one methodology has increased in the recent case studies identified to obtain objective results by combining different MCDM approaches in the HoQ structure. The study carried out by [38] establishes the robust methodology KJ-FCE-FDM-EWM-VIKOR in a two-stage HoQ structure to mitigate sustainable risks in the supply chain sector. In the research article by [83], the DST-AHP&PROMETHEE hybrid methodology was used, also in the field of the supply chain, to obtain the best reconfiguration scenario. In this case, four matrices from the QFD approach were integrated to give a holistic perspective on the processes, activities, roles, and actors related to business processes.

A total of 45% of the 65 articles reviewed used one MCDM methodology in the HoQ structure, and 55% of the studies proposed hybrid MCDM approaches to achieve customer requirements while accurately addressing engineering characteristics. These results show the increasing interest in incorporating more than one MCDM in QFD methodology and it is expected that they will continue to be widely applied in many fields.

4.4. Publications per Year

In 2022, 56 articles related to MCDM methodologies and QFD were published, as it can be seen in Figure 6, which represents a great impact in the field in the last two decades. In the first four months of 2023, 13 studies were identified to achieve supply chain targets [83,87,88] and improvement products [57,89] in post COVID-19 scenarios [87] and also in the energy sector, where an interesting approach was used to address sustainable targets based on a life cycle assessment (LCA) [62] and the creation of roadmaps [82].

The methodologies in HoQ structure are mainly AHP [57], fuzzy AHP [62], and hybrid fuzzy models such as [81,83], PROMETHEE II [89] and TOPSIS [63]. These case studies were applied in the supply chain, sustainability, and design and product development. From these results, it can be expected that more case studies using MCDM in QFD as decision-making tools will appear by the end of the year due to their applicability in a wide range of fields.

4.5. MCDM Methodologies Integrated in QFD

Figure 7 presents an HoQ structure and the MCDM approaches used to improve the QFD method. Most of the articles identified in the present systematic review are focused on the implementation of MCDM methodologies in order to weigh customer importance: AHP, fuzzy AHP [58,59,60,61,62], ANP [76,90], ORESTE [36], fuzzy linguistic preference relations (FLPRs) [42], and q-rung orthopair fuzzy number (q-ROFN) [91].

To rank the WHATs, methodologies such as TOPSIS [67] and the Kano model [92] are also integrated in the QFD methodology. In the roof matrix, the DEMATEL [8] and fuzzy cognitive maps [84] methodologies are taken into consideration to solve contradictions between technical requirements for the customer-oriented product design.

In the customer competitive assessment, methodologies such as AHP and DEMATEL are considered to obtain weight values [8], showing substantial improvements in the construction of the HoQ, to give more certainty in decision-making processes and to the selection of the best option, depending on the specific aim to address.

Some studies ranked the technical requirements, or EC importance, based on the proportional hesitant fuzzy linguistic term sets (PHFLTSs) and the cumulative prospect theory [85].

In the correlation matrix, the MCDMs most successfully implemented to avoid vagueness are fuzzy numbers, the fuzzy linguistic scale, the fuzzy best-worst method, and q-ROFNs-based VIKOR. Finally, to evaluate the HoQ in the ranking section, the neutrosophic operations approach was included in [8] to consider indeterminacy in this section of the HoQ.

4.6. Fields of Application

The fields where the studies selected in this review were developed are shown in Table 4. The most common areas identified in this review were studies related to Planning and Supplier selection. Surprisingly, the limited amount of data referring to the fields of Energy and Environment suggests that, in this sector, MCDM methodologies and QFD are not often applied in the design of renewable technologies. To improve renewable technologies using MCDM in QFD, two articles were identified in the field of solar energy. In [54], the design of a recycled solar air heater was improved using the AHP-QFD approach. The research carried out by [85] ranked engineering characteristics for a solar photovoltaic cell development using proportional hesitant fuzzy linguistic term sets (PHFLTSs).

A comparative scenario of renewable technologies used to analyze the energy poverty was carried out by [93] using fuzzy approaches in QFD to achieve innovation cost targets. In comparison, the case study carried out by [86] assessed the risk and mitigation in the electric sector in Pakistan. These case studies present the importance of using qualitative approaches in the energy sector. However, it is recognized that their use is still limited when decision-making approaches take into account sustainability, engineering, and social aspects. The creation of roadmaps for the energy sector as described in [82], integrating MCDM approaches in QFD, can be useful for forecasting the future of renewable energy technologies.

The articles from the systematic review were classified depending on the aim to achieve. For instance, in the field of Design and product development the research article [42] proposes a sustainable approach for the design of a hybrid electric vehicle based on its life cycle, using MCDM in QFD methodology. In the Planning field, the research shows that MCDM approaches in QFD have important applications in improving strategies in the transport sector in the context of Industry 4.0 [43].

In the field of Manufacturing and materials, MCDM approaches in QFD have become an important tool in decision-making processes, allowing decision makers to select the best processes in various case studies, such as the improvement of welding process selection [59], the performance in metallurgy to reduce the defects in casting and lean tools [94], and manufacturing processes using CNC machines in sheet metal companies [84].

The approach established by [95] incorporates the three pillars of sustainable development to address the stakeholders requirements in the field of manufacturing based on the ‘gate-to-gate’ approach with a sustainability perspective using the AHP methodology.

MCDM methodologies in QFD show wide applicability in many fields due to useful information that they can provide to decision makers. Selecting the most suitable MCDM approach to integrate in QFD will depend on the type of problem to solve, data required, and computing resources.

MATLAB is one of the most used software for computing MCDM methodologies because of its matrix-based language, and it is used in studies such as [104]. LINGO software is used to solve nonlinear models for the F-FUCOM approach and to then apply the results in QFD [35,86], and also to solve fuzzy linear models [79]. The development of algorithms to weigh requirements using BWM is shown in [88]. Furthermore, in the study carried out by [90], the authors used Super Decision software (version 2.10) and QFD templates based on Excel. These findings allow us to establish an overview of the most relevant computing tools used in MCDM and QFD.

Findings of this research show that when the case study incorporates between 10–20 customer and technical requirements, calculations and results from the applied MCDM methodologies must be carried out carefully to avoid mistakes. Moreover, the incorporation of hybrid approaches in QFD adds more inputs to the HoQ structure, which represents a larger matrix, and becomes computationally expensive.

4.7. MCDM Methodologies for ORE Technologies Development

Research articles were found related to MCDM methodologies being used as a tool in the early stages of a project in offshore energy. The case study carried out by [105] describes how the hybrid AHP-PROMETHEE II approach was used to select the optimal location for an offshore wind power station. In [106], the fuzzy SWARA and fuzzy weighted aggregates sum product assessment (WASPAS) methodologies are used to rank sites for the sustainable development of offshore windfarms. The TOPSIS methodology is seen in [107], used to design a floating support structure for a vertical axis wind turbine. MCDM approaches are mainly used in the selection of the best locations for the installation of offshore energy systems for producing electricity. The research article published by [108] offers a state-of-the-art review on the integration of MCDM with LCA in the evaluation of renewable energy technologies. It is clear that the implementation of such methodologies in the offshore renewable sector is lacking, specifically for the tidal and wave sector, and thus the authors propose a framework for use in the development of ORE technologies in the next section.

4.8. A Proposed Framework for Development of ORE Technologies

To reduce uncertainty of stakeholders and decision makers in the ORE field by providing a tool showing the most commonly used MCDM methodologies integrated in the HoQ structure, a proposed framework is presented in Figure 8 with the aim of stimulating developments in offshore energy technologies.

For the customer requirements section, methodologies such as AHP, fuzzy AHP, ANP, TOPSIS, and VIKOR are proposed. Their applicability will depend on the type of CR defined. Although AHP is the approach most often used to weigh customer importance, the findings of this research show that the integration of other approaches to reduce uncertainty in the house of quality can be useful for the QFD structure.

The TOPSIS methodology is proposed to identify the inderpendencies in the customer requirements because of its effectiveness and wide applicability in case studies that need to prioritize elements. ANP is proposed to integrate the CR by constructing a network that allows interdependency among all the elements. In the roof of the HoQ, DEMATEL is seen to be the methodology applied most often to solve technical conflicts among the HOWs. However, approaches such as the theory of inventive thinking (TRIZ), or the failure modes and effects analysis (FMEA), can also be considered for integration in the roof of the HoQ, as in the work of [109], which integrates the TRIZ 40 inventive principles in a tidal project to improve reliability while reducing manufacturing costs.

AHP and DEMATEL approaches are proposed in the HoQ structure to improve the customer competitive assessment in order to obtain weight values from a pairwise comparison among the competitors and compare these values with the customer importance weights [8]. Fuzzy logic theory is suggested to evaluate the correlation matrix between the WHATs and the HOWs, since fuzzy numbers and fuzzy set theory can convert verbal expressions into quantitative information associated with uncertainty evaluation processes [59]. The use of specialized software to rank the results integrating the neutrosophic operations is also recommended, depending on the complexity of the hybrid MCDM-QFD methodology applied.

In order to establish a holistic framework based on the sustainability thinking, the design of ocean technologies must address specific targets aligned with the technical requirements to guarantee the best performance, social acceptability, and environmental and economic indicators. Table 5 presents a proposed framework of energy indicators in a sustainability approach to integrate it in the customer requirements definition in the house of quality structure of [16]. In the energy sector, it is possible that carbon vouchers can be defined as a customer requirement in the HoQ structure so as to promote the sustainable design of ORE technologies and reduce the carbon footprint of their life cycle.

The research carried out by [109] presents an innovative approach used to develop ocean energy systems by integrating QFD, TRIZ, and FMEA in a design tool within the DTOceanPlus suite. This case study shows examples of the definition of objectives and customer requirements for wave energy devices. The authors include annual energy production, capital costs, flexibility of energy supply, and availability as customer requirements. The latter work is considered to be one of the first investigations in the wave sector MCDM methodologies and could be applied to other types of ocean energy technologies. Furthermore, functional requirements are also defined, and design selection decisions are presented in a five-level hierarchical structure that includes lifetime costs, environmental impact, and the relationship between them, providing a holistic analysis. In the present work, an innovative and useful tool that integrates various aspects of sustainable development in ocean technologies is presented.

The proposed framework in Section 4.8 has been designed as a mechanism to design ORE technologies at early stages. It identifies customer requirements when selecting the best scenario based on engineering characteristics as well as economic, social, and environmental aspects. Integrating such decision-making tools could be useful for projects during the 1–6 Technology Readiness Levels (TRLs) from “basic principles observed and reported” to “prototype demonstration in a relevant environment”.

One of the challenges faced when using this tool is the lack of available data to analyze in sustainable assessments given that most of the ocean technology is still at pre-commercial levels. For the environmental assessments, the use of databases to quantify emissions to air, water, and soil is decisive in achieving the design of devices that are sustainable during their life cycle, and they could also be used as environmental indicators to obtain carbon vouchers from the government.

This instinctive tool could contribute to and play a crucial role in creating roadmaps for the sustainable harvesting of ocean resources.

5. Conclusions and Future Work

Having reviewed currently available MCDM approaches in the QFD methodology, it was found that, in 2022, considerably more research articles were published, providing valuable background for future research. These papers concern a wide variety of fields, including education, the supply chain, design and product development, manufacturing and materials, energy, and environment.

There are relatively few case studies that incorporate MCDM approaches as a decision-making tool in the early stages of renewable energy projects. Most of these case studies concern the selection of site locations for offshore technologies. To achieve the net zero emissions targets by 2050, the participation of stakeholders, designers, and specialists is needed if we are to have a greater number of ocean technologies in the energy mix.

The findings of this research show that AHP is the approach most often applied in QFD, 36% of 69 articles, although other MCDM methodologies have recently become attractive as decision-making tools. Fuzzy approaches, such as F-AHP, F-TOPSIS, and F-VIKOR, have been shown to offer important improvements in MCDM methodologies that are used in QFD thanks to fuzzy triangular theory. The use of hybrid methodologies is also expected to be incorporated in the HoQ structure in a robust framework aided by specialized software to promote its use. In the 2022, regarding where the articles were published, five research articles were in the journal Sustainability, four articles in Mathematics, and three in Computers & Industrial Engineering, IEEE Transactions on Engineering Management, and Journal of Intelligent & Fuzzy Systems. There is considerable interest in the use of MCDM methodologies in QFD and the search for new mathematical basis and the integration of specialized software. MCDMs are mainly applied in QFD approaches in planning, design and product development, and in supply chains, as the results allow stakeholders to select the best scenario in a sustainable context.

In developing ORE technologies, a framework that includes AHP, F-AHP, ANP, VIKOR, TOPSIS, DEMATEL, fuzzy logic theory, and neutrosophic operations is proposed as a holist approach that includes environmental issues, social acceptability, and certainty in economic assessments.

Recent case studies published in 2023 such as [110] present the use of the DEA Malmquist model and epsilon-based measure (EBM) to obtain values for the efficiency and inefficiency of lithium-ion producers. In [111], the DEX approach was integrated in the EDAS technique to include data related to operating capacity costs and technical aspects related to manufacturing process; for instance, the precise thickness of workpieces and welding speed to evaluate machines with an efficient and cost-effective approach. These new applications of MCDM approaches could be useful in ORE design tools such as those proposed in this work in order to achieve sustainable targets while considering costs and effectiveness as well as guaranteeing high quality in manufactured technologies.

In the current state of the art, a noticeable knowledge gap exists in the design of sustainable offshore technologies. This gap pertains not only to technical performance but also extends to encompass environmental considerations, economic indicators, and social acceptability. Integrating MCDM methodologies into QFD as decision-making tools, a practice already demonstrated in various industries, holds the potential to facilitate the development of learning curves for ORE technologies. This, in turn, can facilitate the realization of economies of scale within the renewable energy sector.

The researchers behind this study emphasize the adaptability of the house of quality (HoQ) structure, which can effectively accommodate a diverse array of assessments. These assessments encompass a range of life cycle assessment (LCA) studies, covering perspectives such as cradle-to-grave, cradle-to-gate, gate-to-gate, and cradle-to-cradle, with a pronounced focus on the principles of the circular economy. Moreover, the HoQ framework addresses a spectrum of technical considerations, including aspects like efficiency, reliability, and manufacturing costs. Beyond these technical facets, the framework extends its scope to encompass evaluations of potential deployment sites, formulation of effective installation strategies, and the integration of social studies aimed at fostering the ethical use of renewable technologies. Amidst the pursuit of sustainable strategies for the advancement of ocean renewable energy (ORE) technologies, a compelling global challenge emerges: the current lack of incentives provided by international organizations to proactively contribute to the monumental cause of transitioning our energy landscape.

The integration of specialized software, machine learning, and AI to create an instinctive sustainable tool for design could help in assessing different type of studies at the same time and using their results in the house of quality structure easily.

The framework introduced in this study is anticipated to facilitate the development of environmentally sound designs for hydrokinetic turbines and the formulation of strategic pathways within the offshore renewable energy sector.

Author Contributions

Conceptualization, S.G.-O. and G.V.-G.; methodology, S.G.-O. and G.V.-G.; writing—original draft preparation, all the authors.; writing—review and editing, all the authors; funding acquisition, S.O.-S. and R.S. All authors have read and agreed to the published version of the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Stages of this research.

Figure 2. MCDM methodologies in combination with QFD considered in search query of this systematic review.

Figure 3. PRISMA methodology of this systematic review.

Figure 4. Distribution of MCDM methodologies in QFD.

Figure 5. Type of MCDM approaches in QFD.

Figure 6. Publications 2004 to 2021 [6] and 2022 identified in this research.

Figure 7. MCDM methodologies in QFD identified based on the HoQ structure.

Figure 8. Proposed framework for the development of ORE technologies using MCDM in the HoQ structure.

Table 1

MCDM methodologies.

Methodology	Use	References
AHP	Pairwise comparison to generate relative ratio scale of measurement through normalization.	[10,28]
ELECTRE	To outrank relationships throughout pairwise comparison of the alternatives under theestablished criteria separately.	[11,29]
TOPSIS	To choose the best option according to the shortest distance from the ideal solution.	[12,30]
VIKOR	To evaluate and rank alternatives sets of conflicting criteria.	[31,32]
DEMATEL	To analyze the cause-and-effect relationships among components of a system. It is also useful to identify interdependence among factors.	[20,21]
PROMETHEE	To outrank alternatives according to the defined criteria. There is no need ofindependence among the criteria or attributes.	[22,33]
ANP	Pairwise comparison to rank alternatives considering all types of dependencies and interdependencies.	[24,34]

Table 2

Keywords searched.

Topic	Subordinate
“Quality Function Deployment and Multicriteria DecisionMaking”	OR “Quality Function Deployment and Multi-Criteria Decision Making” OR “QFD and Multicriteria Decision Making” OR “QFD and Multi-criteria Decision Making” OR “QFD and MCDM”.
“Quality Function Deployment and Analytic HierarchyProcess”	OR “Quality Function Deployment and AHP” OR “QFD and Analytic Hierarchy Process” OR “QFD and AHP”.
“Quality Function Deployment and Elimination and Choice Translating Reality”	OR “Quality Function Deployment and ELECTRE” OR “QFD and Elimination and Choice Translating Reality” OR “QFD and ELECTRE”.
“Quality Function Deployment and Technique for OrderPreference by Similarity toIdeal Solution”	OR “Quality Function Deployment and TOPSIS” OR “QFD and Technique for Order Preference bySimilarity to Ideal Solution” OR “QFD and TOPSIS”.
“Quality Function Deployment and VlseKriterijumska Optimizacija I Kompromisno Resenje”	OR “Quality Function Deployment and VIKOR” OR “QFD and VlseKriterijumska Optimizacija I Kompromisno Resenje” OR “QFD and VIKOR”.
“Quality Function Deployment and Decision Making Trial and Evaluation Laboratory”	OR “Quality Function Deployment and Decision-Making Trial and Evaluation Laboratory” OR “QFD and Decision Making Trial and Evaluation Laboratory” OR “QFD and Decision-Making Trial andEvaluation Laboratory” OR “Quality Function Deployment and DEMATEL” OR “QFD and DEMATEL”.
“Quality Function Deployment and Preference Ranking Organization Method for Enrichmentof Evaluations”	OR “Quality Function Deployment and PROMETHEE” OR “QFD and Preference RankingOrganization Method for Enrichment of Evaluations” OR “QFD and PROMETHEE”.
“Quality Function Deployment and Analytic Network Process”	OR “Quality Function Deployment and ANP” OR “QFD and Analytic”.

Table 3

Articles published by journals.

Journal	Publications
APPLIED SCIENCES	1
APPLIED SOFT COMPUTING	2
ARABIAN JOURNAL FOR SCIENCE AND ENGINEERING	1
CASE STUDIES ON TRANSPORT POLICY	1
COMPLEX & INTELLIGENT SYSTEMS	1
COMPUTERS & INDUSTRIAL ENGINEERING	3
CONCURRENT ENGINEERING	1
DECISION SUPPORT SYSTEMS	1
EGE ACADEMIC REVIEW	1
ENERGIES	1
ENERGY	1
ENERGY SOURCES PART A-RECOVERY UTILIZATION AND ENVIRONMENTAL EFFECTS	1
ENGINEERING MANAGEMENT JOURNAL	1
ENVIRONMENT DEVELOPMENT AND SUSTAINABILITY	1
EXPERT SYSTEMS WITH APPLICATIONS	1
FINANCIAL INNOVATION	1
IEEE ACCESS	2
IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT	3
INFORMATICA	1
INTERNATIONAL JOURNAL OF CRITICAL INFRASTRUCTURE PROTECTION	1
INTERNATIONAL JOURNAL OF ENERGY RESEARCH	1
INTERNATIONAL JOURNAL OF FUZZY SYSTEMS	1
INTERNATIONAL JOURNAL OF INFORMATION TECHNOLOGY & DECISION MAKING	1
INTERNATIONAL JOURNAL OF INNOVATION AND TECHNOLOGY MANAGEMENT	1
INTERNATIONAL JOURNAL OF LEAN SIX SIGMA	1
INTERNATIONAL JOURNAL OF LOGISTICS MANAGEMENT	1
INTERNATIONAL JOURNAL OF PRODUCTION ECONOMICS	1
INTERNATIONAL JOURNAL OF QUALITY & RELIABILITY MANAGEMENT	1
INTERNATIONAL JOURNAL OF SYSTEMS SCIENCE-OPERATIONS & LOGISTICS	1
JOURNAL OF CLEANER PRODUCTION	1
JOURNAL OF CONTROL AND DECISION	1
JOURNAL OF INDUSTRIAL INFORMATION INTEGRATION	1
JOURNAL OF INFORMATION & KNOWLEDGE MANAGEMENT	1
JOURNAL OF INTELLIGENT & FUZZY SYSTEMS	3
JOURNAL OF MARINE SCIENCE AND ENGINEERING	1
JOURNAL OF THE OPERATIONAL RESEARCH SOCIETY	1
MATHEMATICAL PROBLEMS IN ENGINEERING	2
MATHEMATICS	4
NEURAL COMPUTING & APPLICATIONS	1
PLOS ONE	1
PRODUCTION AND MANUFACTURING RESEARCH-AN OPEN ACCESS JOURNAL	1
PRODUCTION ENGINEERING ARCHIVES	1
SIGMA JOURNAL OF ENGINEERING AND NATURAL SCIENCES	1
SN APPLIED SCIENCES	1
SOCIO-ECONOMIC PLANNING SCIENCES	1
SOFT COMPUTING	2
SUSTAINABILITY	5
SYSTEMS	1
TECHNOLOGICAL FORECASTING AND SOCIAL CHANGE	1
TECHNOLOGY ANALYSIS & STRATEGIC MANAGEMENT	1
TQM JOURNAL	1
UNIVERSAL ACCESS IN THE INFORMATION SOCIETY	1

Table 4

Fields of application identified.

Field	Studies
Education	[50,79]
Energy	[54,68,82,85,86,93]
Environment	[69]
Manufacturing and materials	[59,84,94,95]
Planning	[39,43,48,52,53,55,56,60,65,66,70,71,81,83,90,96,97,98,99]
Design and product development	[8,36,42,51,57,61,62,64,67,72,77,78,80,89,91,92,100,101,102]
Supplier selection	[35,37,38,40,45,46,47,49,58,63,73,74,75,76,87,88,103,104]

Table 5

Proposed framework of energy indicators.

Field	Indicators
Environmental	NOx emissionCO2 emissionCO emissionSO2 emission
Social	Social acceptabilityJob creationSocial benefits
Economic	Net present valueInvestment costEquivalent annual costLevelized cost of energy (LCOE)

References

1. Triantaphyllou, E. Multi-Criteria Decision Making Methods. Multi-Criteria Decision Making Methods: A Comparative Study; Triantaphyllou, E. Springer US: Boston, MA, USA, 2000; pp. 5-21.

2. Chan, L.-K.; Wu, M.-L. Quality function deployment: A literature review. Eur. J. Oper. Res.; 2002; 143, pp. 463-497. [DOI: https://dx.doi.org/10.1016/S0377-2217(02)00178-9]

3. Pavan, M.; Todeschini, R. 1.19—Multicriteria Decision-Making Methods. Comprehensive Chemometrics; Brown, S.D.; Tauler, R.; Walczak, B. Elsevier: Oxford, UK, 2009; pp. 591-629.

4. Samsó, R.; Crespin, J.; García-Olivares, A.; Solé, J. Examining the Potential of Marine Renewable Energy: A Net Energy Perspective. Sustainability; 2023; 15, 8050. [DOI: https://dx.doi.org/10.3390/su15108050]

5. Trivedi, A.; Trivedi, V.; Pandey, K.K.; Chichi, O. An interpretive model to assess the barriers to ocean energy toward blue economic development in India. Renew. Energy; 2023; 211, pp. 822-830. [DOI: https://dx.doi.org/10.1016/j.renene.2023.05.046]

6. Hariri, A.; Domingues, P.; Sampaio, P. Integration of multi-criteria decision-making approaches adapted for quality function deployment: An analytical literature review and future research agenda. Int. J. Qual. Reliab. Manag.; 2023; ahead-of-print [DOI: https://dx.doi.org/10.1108/IJQRM-02-2022-0058]

7. Huang, J.; Mao, L.-X.; Liu, H.-C.; Song, M.-s. Quality function deployment improvement: A bibliometric analysis and literature review. Qual. Quant.; 2022; 56, pp. 1347-1366. [DOI: https://dx.doi.org/10.1007/s11135-021-01179-7]

8. Karasan, A.; Ilbahar, E.; Cebi, S.; Kahraman, C. Customer-oriented product design using an integrated neutrosophic AHP & DEMATEL & QFD methodology. Appl. Soft Comput.; 2022; 118, 108445. [DOI: https://dx.doi.org/10.1016/j.asoc.2022.108445]

9. Saaty, T.L.; Vargas, L.G. How to Make a Decision. Models, Methods, Concepts & Applications of the Analytic Hierarchy Process; Saaty, T.L.; Vargas, L.G. Springer US: Boston, MA, USA, 2012; pp. 1-21.

10. Saaty, T.L. The Modern Science of Multicriteria Decision Making and Its Practical Applications: The AHP/ANP Approach. Oper. Res.; 2013; 61, pp. 1101-1118. [DOI: https://dx.doi.org/10.1287/opre.2013.1197]

11. Govindan, K.; Jepsen, M.B. ELECTRE: A comprehensive literature review on methodologies and applications. Eur. J. Oper. Res.; 2016; 250, pp. 1-29. [DOI: https://dx.doi.org/10.1016/j.ejor.2015.07.019]

12. Çelikbilek, Y.; Tüysüz, F. An in-depth review of theory of the TOPSIS method: An experimental analysis. J. Manag. Anal.; 2020; 7, pp. 281-300. [DOI: https://dx.doi.org/10.1080/23270012.2020.1748528]

13. Deng, H.; Yeh, C.-H.; Willis, R.J. Inter-company comparison using modified TOPSIS with objective weights. Comput. Oper. Res.; 2000; 27, pp. 963-973. [DOI: https://dx.doi.org/10.1016/S0305-0548(99)00069-6]

14. Chakraborty, S. TOPSIS and Modified TOPSIS: A comparative analysis. Decis. Anal. J.; 2022; 2, 100021. [DOI: https://dx.doi.org/10.1016/j.dajour.2021.100021]

15. Gul, M.; Celik, E.; Aydin, N.; Gumus, A.T.; Guneri, A.F. A state of the art literature review of VIKOR and its fuzzy extensions on applications. Appl. Soft Comput.; 2016; 46, pp. 60-89. [DOI: https://dx.doi.org/10.1016/j.asoc.2016.04.040]

16. Kaya, T.; Kahraman, C. Multicriteria renewable energy planning using an integrated fuzzy VIKOR & AHP methodology: The case of Istanbul. Energy; 2010; 35, pp. 2517-2527. [DOI: https://dx.doi.org/10.1016/j.energy.2010.02.051]

17. Alinezhad, A.; Khalili, J. VIKOR Method. New Methods and Applications in Multiple Attribute Decision Making (MADM); Alinezhad, A.; Khalili, J. Springer International Publishing: Cham, Switzerland, 2019; pp. 23-27.

18. Yazdi, M.; Khan, F.; Abbassi, R.; Rusli, R. Improved DEMATEL methodology for effective safety management decision-making. Saf. Sci.; 2020; 127, 104705. [DOI: https://dx.doi.org/10.1016/j.ssci.2020.104705]

19. Alinezhad, A.; Khalili, J. DEMATEL Method. New Methods and Applications in Multiple Attribute Decision Making (MADM); Alinezhad, A.; Khalili, J. Springer International Publishing: Cham, Switzerland, 2019; pp. 103-108.

20. Lee, H.-S.; Tzeng, G.-H.; Yeih, W.; Wang, Y.-J.; Yang, S.-C. Revised DEMATEL: Resolving the Infeasibility of DEMATEL. Appl. Math. Model.; 2013; 37, pp. 6746-6757. [DOI: https://dx.doi.org/10.1016/j.apm.2013.01.016]

21. Si, S.-L.; You, X.-Y.; Liu, H.-C.; Zhang, P. DEMATEL Technique: A Systematic Review of the State-of-the-Art Literature on Methodologies and Applications. Math. Probl. Eng.; 2018; 2018, 3696457. [DOI: https://dx.doi.org/10.1155/2018/3696457]

22. Behzadian, M.; Kazemzadeh, R.B.; Albadvi, A.; Aghdasi, M. PROMETHEE: A comprehensive literature review on methodologies and applications. Eur. J. Oper. Res.; 2010; 200, pp. 198-215. [DOI: https://dx.doi.org/10.1016/j.ejor.2009.01.021]

23. Alinezhad, A.; Khalili, J. PROMETHEE I-II-III Methods. New Methods and Applications in Multiple Attribute Decision Making (MADM); Alinezhad, A.; Khalili, J. Springer International Publishing: Cham, Switzerland, 2019; pp. 29-39.

24. Sipahi, S.; Timor, M. The analytic hierarchy process and analytic network process: An overview of applications. Manag. Decis.; 2010; 48, pp. 775-808. [DOI: https://dx.doi.org/10.1108/00251741011043920]

25. Kheybari, S.; Rezaie, F.M.; Farazmand, H. Analytic network process: An overview of applications. Appl. Math. Comput.; 2020; 367, 124780. [DOI: https://dx.doi.org/10.1016/j.amc.2019.124780]

26. Saaty, T.L. Fundamentals of the analytic network process—Dependence and feedback in decision-making with a single network. J. Syst. Sci. Syst. Eng.; 2004; 13, pp. 129-157. [DOI: https://dx.doi.org/10.1007/s11518-006-0158-y]

27. Alinezhad, A.; Khalili, J. ANP Method. New Methods and Applications in Multiple Attribute Decision Making (MADM); Alinezhad, A.; Khalili, J. Springer International Publishing: Cham, Switzerland, 2019; pp. 115-125.

28. Saaty, T.L. How to make a decision: The analytic hierarchy process. Eur. J. Oper. Res.; 1990; 48, pp. 9-26. [DOI: https://dx.doi.org/10.1016/0377-2217(90)90057-I]

29. Figueira, J.; Roy, B. Determining the weights of criteria in the ELECTRE type methods with a revised Simos’ procedure. Eur. J. Oper. Res.; 2002; 139, pp. 317-326. [DOI: https://dx.doi.org/10.1016/S0377-2217(01)00370-8]

30. Tadic, D.; Gumus, A.T.; Arsovski, S.; Aleksic, A.; Stefanovic, M. An evaluation of quality goals by using fuzzy AHP and fuzzy TOPSIS methodology. J. Intell. Fuzzy Syst.; 2013; 25, pp. 547-556. [DOI: https://dx.doi.org/10.3233/IFS-120659]

31. Mardani, A.; Zavadskas, E.K.; Govindan, K.; Amat Senin, A.; Jusoh, A. VIKOR Technique: A Systematic Review of the State of the Art Literature on Methodologies and Applications. Sustainability; 2016; 8, 37. [DOI: https://dx.doi.org/10.3390/su8010037]

32. Wan, S.-P.; Wang, Q.-Y.; Dong, J.-Y. The extended VIKOR method for multi-attribute group decision making with triangular intuitionistic fuzzy numbers. Knowl. Based Syst.; 2013; 52, pp. 65-77. [DOI: https://dx.doi.org/10.1016/j.knosys.2013.06.019]

33. Mateo, J.R.S.C. PROMETHEE. Multi Criteria Analysis in the Renewable Energy Industry; San Cristóbal Mateo, J.R. Springer London: London, UK, 2012; pp. 23-32.

34. Saaty, T.L. The Analytic Network Process. Decision Making with the Analytic Network Process: Economic, Political, Social and Technological Applications with Benefits, Opportunities, Costs and Risks; Saaty, T.L.; Vargas, L.G. Springer US: Boston, MA, USA, 2006; pp. 1-26.

35. Khan, F.; Ali, Y. Implementation of the circular supply chain management in the pharmaceutical industry. Environ. Dev. Sustain.; 2022; 24, pp. 13705-13731. [DOI: https://dx.doi.org/10.1007/s10668-021-02007-6] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/35035276]

36. Shi, H.; Mao, L.-X.; Li, K.; Wang, X.-H.; Liu, H.-C. Engineering Characteristics Prioritization in Quality Function Deployment Using an Improved ORESTE Method with Double Hierarchy Hesitant Linguistic Information. Sustainability; 2022; 14, 9771. [DOI: https://dx.doi.org/10.3390/su14159771]

37. Hsu, C.-H.; Yu, R.-Y.; Chang, A.-Y.; Liu, W.-L.; Sun, A.-C. Applying Integrated QFD-MCDM Approach to Strengthen Supply Chain Agility for Mitigating Sustainable Risks. Mathematics; 2022; 10, 552. [DOI: https://dx.doi.org/10.3390/math10040552]

38. Hsu, C.-H.; Li, M.-G.; Zhang, T.-Y.; Chang, A.-Y.; Shangguan, S.-Z.; Liu, W.-L. Deploying Big Data Enablers to Strengthen Supply Chain Resilience to Mitigate Sustainable Risks Based on Integrated HOQ-MCDM Framework. Mathematics; 2022; 10, 1233. [DOI: https://dx.doi.org/10.3390/math10081233]

39. Zhang, X.; Su, J.; Herrera-Viedma, E. A decision support model for estimating participation-oriented designs of crowdsourcing platforms based on quality function deployment. Expert Syst. Appl.; 2022; 202, 117308. [DOI: https://dx.doi.org/10.1016/j.eswa.2022.117308]

40. Karuppiah, K.; Sankaranarayanan, B.; Ali, S.M. A Novel Quality Function Deployment Based Integrated Framework for Improving Supply Chain Sustainability. Eng. Manag. J.; 2022; 35, pp. 285-298. [DOI: https://dx.doi.org/10.1080/10429247.2022.2097575]

41. Zhou, J.; Shen, Y.; Pantelous, A.A.; Liu, Y. Quality Function Deployment: A Bibliometric-Based Overview. IEEE Trans. Eng. Manag.; 2022; 56, pp. 1-22. [DOI: https://dx.doi.org/10.1109/TEM.2022.3146534]

42. Kang, X.; Zhu, Q. Integrated fuzzy linguistic preference relations approach and fuzzy Quality Function Deployment to the sustainable design of hybrid electric vehicles. Concurr. Eng.; 2022; 30, pp. 367-381. [DOI: https://dx.doi.org/10.1177/1063293X221117291]

43. Torkayesh, A.E.; Yazdani, M.; Ribeiro-Soriano, D. Analysis of industry 4.0 implementation in mobility sector: An integrated approach based on QFD, BWM, and stratified combined compromise solution under fuzzy environment. J. Ind. Inf. Integr.; 2022; 30, 100406. [DOI: https://dx.doi.org/10.1016/j.jii.2022.100406]

44. Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E. et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ; 2021; 372, n71. [DOI: https://dx.doi.org/10.1136/bmj.n71] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33782057]

45. De Araújo Souza, S.M.O.; de Oliveira Fontes, C.H.; Freires, F.G.M. Use of QFD to Prioritize Requirements Needed for Supplier Selection in an O&G Project. Proceedings of the Industrial Engineering and Operations Management; Istanbul, Turkey, 7 March 2022; Springer: Cham, Switzerland, 2022; pp. 169-180.

46. Yu, R.-Y.; Hsu, C.-H.; Sun, A.-C. Improving Supply Chain Agility to Reduce Enterprise Risk Based on QFD-MADM. Proceedings of the Advances in Intelligent Systems and Computing; Athens, Greece, 29–31 August 2022; Springer: Singapore, 2022; pp. 15-23.

47. Zeng, J.-Y.; Hsu, C.-H.; Chen, X. Using Agility to Reduce the Bullwhip Effect of Supply Chains. Proceedings of the Advances in Intelligent Systems and Computing; Athens, Greece, 29–31 August 2022; Springer: Singapore, 2022; pp. 43-51.

48. Wu, X.; Zhang, Z. The Post-pandemic Era Study on the Design of High-speed Railway Seats. Proceedings of the Design, User Experience, and Usability: Design Thinking and Practice in Contemporary and Emerging Technologies; Virtual Event, 26 June 2022; Springer: Cham, Switzerland, 2022; pp. 265-278.

49. Erbay, H.; Yıldırım, N. Combined Technology Selection Model for Digital Transformation in Manufacturing: A Case Study From the Automotive Supplier Industry. Int. J. Innov. Technol. Manag.; 2022; 19, 2250023. [DOI: https://dx.doi.org/10.1142/S0219877022500237]

50. Ocak, Z. An integrated approach to identify engineering student requirements. Sigma J. Eng. Nat. Sci. Sigma Mühendislik Ve Fen Bilim. Derg.; 2022; 40, pp. 568-576. [DOI: https://dx.doi.org/10.14744/sigma.2022.00061]

51. Zhu, T.-L.; Li, Y.-J.; Wu, C.-J.; Yue, H.; Zhao, Y.-Q. Research on the Design of Surgical Auxiliary Equipment Based on AHP, QFD, and PUGH Decision Matrix. Math. Probl. Eng.; 2022; 2022, 4327390. [DOI: https://dx.doi.org/10.1155/2022/4327390]

52. Zhou, J.; Tan, R.; Lin, H.-C. Development of an integrated conceptual path model for a smart elderly care information system. Univers. Access Inf. Soc.; 2022; 22, pp. 785-810. [DOI: https://dx.doi.org/10.1007/s10209-022-00879-7]

53. Murugan, M.; Marisamynathan, S. Elucidating the Indian customers requirements for electric vehicle adoption: An integrated analytical hierarchy process—Quality function deployment approach. Case Stud. Transp. Policy; 2022; 10, pp. 1045-1057. [DOI: https://dx.doi.org/10.1016/j.cstp.2022.03.017]

54. El Badaoui, M.; Touzani, A. AHP QFD methodology for a recycled solar collector. Prod. Eng. Arch.; 2022; 28, pp. 30-39. [DOI: https://dx.doi.org/10.30657/pea.2022.28.04]

55. Pusztai, L.P.; Nagy, L.; Budai, I. Selection of Production Reliability Indicators for Project Simulation Model. Appl. Sci.; 2022; 12, 5012. [DOI: https://dx.doi.org/10.3390/app12105012]

56. Lestari, F.; Mawardi, M.; Yola, M.; Muda, Y.; Abdul Hamid, A.B. A model for assessment of Halal Good Manufacturing Practice in meat industry. Prod. Manuf. Res.; 2022; 10, pp. 666-695. [DOI: https://dx.doi.org/10.1080/21693277.2022.2117742]

57. Haghshenas Gorgani, H.; Jahazi, A.; Jahantigh Pak, A.; Shabani, S. A hybrid algorithm for adjusting the input parameters of the wirecut EDM machine in order to obtain maximum customer satisfaction. SN Appl. Sci.; 2022; 5, 37. [DOI: https://dx.doi.org/10.1007/s42452-022-05256-w]

58. Rehman, O.u.; Ali, Y. Enhancing healthcare supply chain resilience: Decision-making in a fuzzy environment. Int. J. Logist. Manag.; 2022; 33, pp. 520-546. [DOI: https://dx.doi.org/10.1108/IJLM-01-2021-0004]

59. Omar, M.; Janada, K.; Soltan, H. FAQT: A Precise System for Welding Process Selection. Int. J. Fuzzy Syst.; 2022; 24, pp. 1605-1618. [DOI: https://dx.doi.org/10.1007/s40815-021-01219-2]

60. Baskir, M.B. A novel belief-based QFD-AHP model in interval type-2 fuzzy environment for lean after-sales service in automotive industry. Int. J. Lean Six Sigma; 2023; 14, pp. 653-678. [DOI: https://dx.doi.org/10.1108/IJLSS-03-2022-0055]

61. Wang, N.; Shi, C.; Kang, X. Design of a Disinfection and Epidemic Prevention Robot Based on Fuzzy QFD and the ARIZ Algorithm. Sustainability; 2022; 14, 16341. [DOI: https://dx.doi.org/10.3390/su142416341]

62. Jagoda, S.U.M.; Gamage, J.R.; Karunathilake, H.P. Environmentally sustainable plastic food packaging: A holistic life cycle thinking approach for design decisions. J. Clean. Prod.; 2023; 400, 136680. [DOI: https://dx.doi.org/10.1016/j.jclepro.2023.136680]

63. Sharma, J.; Tripathy, B.B. An integrated QFD and fuzzy TOPSIS approach for supplier evaluation and selection. TQM J.; 2023; ahead-of-print [DOI: https://dx.doi.org/10.1108/TQM-09-2022-0295]

64. Fang, H.; Li, J.; Song, W. A New Method for Quality Function Deployment Based on Rough Cloud Model Theory. IEEE Trans. Eng. Manag.; 2022; 69, pp. 2842-2856. [DOI: https://dx.doi.org/10.1109/TEM.2020.3020339]

65. Wang, Y.-J.; Liu, L.-J.; Han, T.-C. Interval-Valued Fuzzy Multi-Criteria Decision-Making with Dependent Evaluation Criteria for Evaluating Service Performance of International Container Ports. J. Mar. Sci. Eng.; 2022; 10, 991. [DOI: https://dx.doi.org/10.3390/jmse10070991]

66. Cho, J.; Chae, M. Systematic Approach of TOPSIS Decision-Making for Construction Method Based on Risk Reduction Feedback of Extended QFD-FMEA. Math. Probl. Eng.; 2022; 2022, 1458599. [DOI: https://dx.doi.org/10.1155/2022/1458599]

67. Hameed, A.Z.; Kandasamy, J.; Aravind Raj, S.; Baghdadi, M.A.; Shahzad, M.A. Sustainable Product Development Using FMEA ECQFD TRIZ and Fuzzy TOPSIS. Sustainability; 2022; 14, 14345. [DOI: https://dx.doi.org/10.3390/su142114345]

68. Li, W.; Yüksel, S.; Dinçer, H. Understanding the financial innovation priorities for renewable energy investors via QFD-based picture fuzzy and rough numbers. Financ. Innov.; 2022; 8, 67. [DOI: https://dx.doi.org/10.1186/s40854-022-00372-3]

69. Efe, B.; Efe, Ö.F.; Ishizaka, A. A model proposal to examine the effects of ships to marine pollution in terms of internal and external factors. Soft Comput.; 2022; 26, pp. 2121-2134. [DOI: https://dx.doi.org/10.1007/s00500-021-06626-z]

70. Xu, X.-G.; Zhang, L.; Mao, L.-X.; Li, K. New Approach for Quality Function Deployment Using an Extended CoCoSo Method with Spherical Fuzzy Sets. Systems; 2022; 10, 253. [DOI: https://dx.doi.org/10.3390/systems10060253]

71. Cinar, U.; Cebi, S. A novel approach to assess occupational risks and prevention of hazards: The house of safety & prevention. J. Intell. Fuzzy Syst.; 2022; 42, pp. 517-528. [DOI: https://dx.doi.org/10.3233/jifs-219208]

72. Li, Y.-H.; Yue, S.-T.; Zheng, J.; Wang, W. Customer-oriented product design: An integrated decision framework with sentiment analysis and optimisation model. J. Control Decis.; 2022; pp. 1-15. [DOI: https://dx.doi.org/10.1080/23307706.2022.2146607]

73. Erol, I.; Murat Ar, I.; Peker, I.; Searcy, C. Alleviating the Impact of the Barriers to Circular Economy Adoption Through Blockchain: An Investigation Using an Integrated MCDM-based QFD With Hesitant Fuzzy Linguistic Term Sets. Comput. Ind. Eng.; 2022; 165, 107962. [DOI: https://dx.doi.org/10.1016/j.cie.2022.107962]

74. Finger, G.S.W.; Lima-Junior, F.R. A hesitant fuzzy linguistic QFD approach for formulating sustainable supplier development programs. Int. J. Prod. Econ.; 2022; 247, 108428. [DOI: https://dx.doi.org/10.1016/j.ijpe.2022.108428]

75. Hsu, C.H.; Zeng, J.Y.; Chang, A.Y.; Cai, S.Q. Deploying Industry 4.0 Enablers to Strengthen Supply Chain Resilience to Mitigate Ripple Effects: An Empirical Study of Top Relay Manufacturer in China. IEEE Access; 2022; 10, pp. 114829-114855. [DOI: https://dx.doi.org/10.1109/ACCESS.2022.3215620]

76. Peng, Z.; He, L.; Xie, Y.; Song, W.; Liu, J.; Ming, X.; Goh, M. A Pythagorean fuzzy ANP-QFD-Grey relational analysis approach to prioritize design requirements of sustainable supply chain. J. Intell. Fuzzy Syst.; 2022; 42, pp. 3893-3907. [DOI: https://dx.doi.org/10.3233/JIFS-212131]

77. Haktanir, E.; Kahraman, C. New Product Design Using Chebyshev’s Inequality Based Interval-Valued Intuitionistic Z-Fuzzy QFD Method. Informatica; 2022; 33, pp. 1-33. [DOI: https://dx.doi.org/10.15388/22-INFOR476]

78. Yazdani, M.; Pamucar, D.; Erdmann, A.; Toro-Dupouy, L. Resilient sustainable investment in digital education technology: A stakeholder-centric decision support model under uncertainty. Technol. Forecast. Soc. Chang.; 2023; 188, 122282. [DOI: https://dx.doi.org/10.1016/j.techfore.2022.122282]

79. Pradhan, P.; Shabbiruddin,; Pradhan, S. Selection of electric vehicle using integrated Fuzzy-MCDM approach with analysis on challenges faced in hilly terrain. Energy Sources Part A Recovery Util. Environ. Eff.; 2022; 44, pp. 2651-2673. [DOI: https://dx.doi.org/10.1080/15567036.2022.2056665]

80. Liu, P.; Zhang, K.; Dong, X.; Wang, P. A Big Data-Kano and SNA-CRP Based QFD Model: Application to Product Design Under Chinese New E-commerce Model. IEEE Trans. Eng. Manag.; 2022; pp. 1-15. [DOI: https://dx.doi.org/10.1109/TEM.2022.3227094]

81. Kou, G.; Dinçer, H.; Yüksel, S.; Alotaibi, F.S. Imputed Expert Decision Recommendation System for QFD-based Omnichannel Strategy Selection for Financial Services. Int. J. Inf. Technol. Decis. Mak.; 2023; 2330003. [DOI: https://dx.doi.org/10.1142/S0219622023300033]

82. Lu, S.; Zhou, J.; Ren, J. Alleviating Energy Poverty through Renewable Energy Technology: An Investigation Using a Best-Worst Method-Based Quality Function Deployment Approach with Interval-Valued Intuitionistic Fuzzy Numbers. Int. J. Energy Res.; 2023; 2023, 8358799. [DOI: https://dx.doi.org/10.1155/2023/8358799]

83. Zidi, S.; Ben Soussia, T.; Hamani, N.; Benaissa, M.; Kermad, L. A new approach for business process reconfiguration under uncertainty using Dempster-Shafer Theory. Int. J. Syst. Sci. Oper. Logist.; 2023; 10, 2017062. [DOI: https://dx.doi.org/10.1080/23302674.2021.2017062]

84. Emel, G.G.; Petriçli, G.; Kayguluoğlu, C. Integrating Quality Function Deployment with Fuzzy Cognitive Maps for Resolving Correlation Issues in the Roof Matrix. Ege Akad. Bakis (Ege Acad. Rev.); 2022; pp. 117-138. [DOI: https://dx.doi.org/10.21121/eab.787075]

85. Yang, Q.; Chen, Z.-S.; Chan, C.Y.P.; Pedrycz, W.; Martínez, L.; Skibniewski, M.J. Large-scale group decision-making for prioritizing engineering characteristics in quality function deployment under comparative linguistic environment. Appl. Soft Comput.; 2022; 127, 109359. [DOI: https://dx.doi.org/10.1016/j.asoc.2022.109359]

86. Rehman, O.u.; Ali, Y.; Sabir, M. Risk assessment and mitigation for electric power sectors: A developing country’s perspective. Int. J. Crit. Infrastruct. Prot.; 2022; 36, 11. [DOI: https://dx.doi.org/10.1016/j.ijcip.2021.100507]

87. Ocampo, L.; Aro, J.L.; Evangelista, S.S.; Maturan, F.; Atibing, N.M.; Yamagishi, K.; Selerio, E. Synthesis of strategies in post-COVID-19 public sector supply chains under an intuitionistic fuzzy environment. Socio-Econ. Plan. Sci.; 2023; 85, 101340. [DOI: https://dx.doi.org/10.1016/j.seps.2022.101340]

88. Chang, J.-P.; Chen, Z.-S.; Wang, X.-J.; Martínez, L.; Pedrycz, W.; Skibniewski, M.J. Requirement-driven sustainable supplier selection: Creating an integrated perspective with stakeholders’ interests and the wisdom of expert crowds. Comput. Ind. Eng.; 2023; 175, 20. [DOI: https://dx.doi.org/10.1016/j.cie.2022.108903]

89. Lian, X.; Hou, L.; Zhang, W.; Yan, H.; Liu, Y. Identifying risky components of display products for redesign considering user attention and failure causality. Soft Comput.; 2023; 27, pp. 2921-2942. [DOI: https://dx.doi.org/10.1007/s00500-022-07660-1]

90. Wudhikarn, R.; Pongpatcharatorntep, D. An Improved Intellectual Capital Management Method for Selecting and Prioritizing Intangible-Related Aspects: A Case Study of Small Enterprise in Thailand. Mathematics; 2022; 10, 626. [DOI: https://dx.doi.org/10.3390/math10040626]

91. Efe, Ö.F.; Efe, B. A decision support model based on q-rung orthopair fuzzy number for glove design application. Neural Comput. Appl.; 2022; 34, pp. 12695-12708. [DOI: https://dx.doi.org/10.1007/s00521-022-07118-3]

92. Tandiono, Y.; Rau, H. An Enhanced Model Using the Kano Model, QFDE, and TRIZ with a Component-Based Approach for Sustainable and Innovative Product Design. Sustainability; 2022; 15, 527. [DOI: https://dx.doi.org/10.3390/su15010527]

93. Zhang, Y.; Zhang, Y.; Gong, C.; Dinçer, H.; Yüksel, S. An integrated hesitant 2-tuple Pythagorean fuzzy analysis of QFD-based innovation cost and duration for renewable energy projects. Energy; 2022; 248, 123561. [DOI: https://dx.doi.org/10.1016/j.energy.2022.123561]

94. Senthil Kannan, N.; Parameshwaran, R.; Saravanakumar, P.T.; Kumar, P.M.; Rinawa, M.L. Performance and Quality Improvement in a Foundry Industry using Fuzzy MCDM and Lean Methods. Arab. J. Sci. Eng.; 2022; 47, pp. 15379-15390. [DOI: https://dx.doi.org/10.1007/s13369-022-06627-6]

95. Rasheed, A.; Ion, W. A Novel Approach towards Sustainability Assessment in Manufacturing and Stakeholder’s Role. Sustainability; 2022; 14, 3221. [DOI: https://dx.doi.org/10.3390/su14063221]

96. Jafarzadeh, H.; Heidary-Dahooie, J.; Akbari, P.; Qorbani, A. A project prioritization approach considering uncertainty, reliability, criteria prioritization, and robustness. Decis. Support Syst.; 2022; 156, 17. [DOI: https://dx.doi.org/10.1016/j.dss.2022.113731]

97. Mello, V.G.d.; Kovaleski, J.L.; Zola, F.C.; Lima Junior, F.R.; Aragão, F.V.; Chiroli, D.M.D.G. Proposal of a Fuzzy-QFD model for startup selection. Technol. Anal. Strateg. Manag.; 2022; pp. 1-16. [DOI: https://dx.doi.org/10.1080/09537325.2022.2046725]

98. Reis, L.P.; Fernandes, J.M.; Silva, S.E.; Pereira, A.D.S. Application of Quality Function Deployment as an Integrative Method to Knowledge Management Implementation. J. Inf. Knowl. Manag.; 2022; 21, 2250022. [DOI: https://dx.doi.org/10.1142/S0219649222500228]

99. Xiao, J.; Wang, X.; Zhang, B.; Zhang, H. Consensus-driven methodology to managing diversity and complex linguistic ratings in quality function deployment: An optimization-based approach. J. Oper. Res. Soc.; 2022; pp. 1-22. [DOI: https://dx.doi.org/10.1080/01605682.2022.2129482]

100. Hong, Z.; Feng, Y.; Wang, Y.; Hu, B.; Zhang, Z.; Qiu, H.; Tan, J. Quality characteristic extraction for complex products with multi-granular fuzzy language based on the triple bottom lines of sustainability. Comput. Ind. Eng.; 2022; 167, 107980. [DOI: https://dx.doi.org/10.1016/j.cie.2022.107980]

101. Wang, Z.-Q.; Chen, Z.-S.; Garg, H.; Pu, Y.; Chin, K.-S. An integrated quality-function-deployment and stochastic-dominance-based decision-making approach for prioritizing product concept alternatives. Complex Intell. Syst.; 2022; 8, pp. 2541-2556. [DOI: https://dx.doi.org/10.1007/s40747-022-00681-1]

102. Porto de Lima, B.; da Silva, A.F.; Marins, F.A.S. New hybrid AHP-QFD-PROMETHEE decision-making support method in the hesitant fuzzy environment: An application in packaging design selection. J. Intell. Fuzzy Syst.; 2022; 42, pp. 2881-2897. [DOI: https://dx.doi.org/10.3233/JIFS-201739]

103. Hsu, C.-H.; He, X.; Zhang, T.-Y.; Chang, A.-Y.; Liu, W.-L.; Lin, Z.-Q. Enhancing Supply Chain Agility with Industry 4.0 Enablers to Mitigate Ripple Effects Based on Integrated QFD-MCDM: An Empirical Study of New Energy Materials Manufacturers. Mathematics; 2022; 10, 1635. [DOI: https://dx.doi.org/10.3390/math10101635]

104. Khattak, B.K.; Naseem, A.; Ullah, M.; Imran, M.; El Ferik, S. Incorporating management opinion in green supplier selection model using quality function deployment and interactive fuzzy programming. PLoS ONE; 2022; 17, e0268552. [DOI: https://dx.doi.org/10.1371/journal.pone.0268552]

105. Abdel-Basset, M.; Gamal, A.; Chakrabortty, R.K.; Ryan, M. A new hybrid multi-criteria decision-making approach for location selection of sustainable offshore wind energy stations: A case study. J. Clean. Prod.; 2021; 280, 124462. [DOI: https://dx.doi.org/10.1016/j.jclepro.2020.124462]

106. Hosseini Dehshiri, S.S. New hybrid multi criteria decision making method for offshore windfarm site location in Persian Gulf, Iran. Ocean Eng.; 2022; 256, 111498. [DOI: https://dx.doi.org/10.1016/j.oceaneng.2022.111498]

107. Collu, M.; Brennan, F.P.; Patel, M.H. Conceptual design of a floating support structure for an offshore vertical axis wind turbine: The lessons learnt. Ships Offshore Struct.; 2014; 9, pp. 3-21. [DOI: https://dx.doi.org/10.1080/17445302.2012.698896]

108. Campos-Guzmán, V.; García-Cáscales, M.S.; Espinosa, N.; Urbina, A. Life Cycle Analysis with Multi-Criteria Decision Making: A review of approaches for the sustainability evaluation of renewable energy technologies. Renew. Sustain. Energy Rev.; 2019; 104, pp. 343-366. [DOI: https://dx.doi.org/10.1016/j.rser.2019.01.031]

109. Tunga, I.; Garcia-Teruel, A.; Noble, D.R.; Henderson, J. Addressing European Ocean Energy Challenge: The DTOceanPlus Structured Innovation Tool for Concept Creation and Selection. Energies; 2021; 14, 5988. [DOI: https://dx.doi.org/10.3390/en14185988]

110. Wang, C.-N.; Yang, F.-C.; Vo, N.T.M.; Nguyen, V.T. Enhancing Lithium-Ion Battery Manufacturing Efficiency: A Comparative Analysis Using DEA Malmquist and Epsilon-Based Measures. Batteries; 2023; 9, 317. [DOI: https://dx.doi.org/10.3390/batteries9060317]

111. Wang, C.-N.; Yang, F.-C.; Vo, T.M.; Nguyen, V.T.; Singh, M. Enhancing Efficiency and Cost-Effectiveness: A Groundbreaking Bi-Algorithm MCDM Approach. Appl. Sci.; 2023; 13, 9105. [DOI: https://dx.doi.org/10.3390/app13169105]