A Systematic Review of the Delphi–AHP Method in Analyzing Challenges to Public-Sector Project Procurement and the Supply Chain: A Developing Country’s Perspective

Abstract

The effectiveness of public-sector agricultural development projects in developing countries lies not only in their contributions to agricultural sector growth but also in their contributions to environmental and socioeconomic system growth. As a result, the challenges associated with project procurement and supply chain management need to be carefully analyzed and evaluated. Although there has been reasonable literature on procurement and supply chain management, the limitations include the following: The literature, especially focusing on the analytical methodology, is scarce, as is the case with the developing country public-sector project context. This study, in its own modest way, contributes to this gap. Thus, the goal of this paper is to critically examine the Delphi and/or analytical hierarchy process (AHP), as well as their application and appropriateness in analyzing the challenges in the Bangladesh context, from relevant literature published between 2000 and 2019. A systematic review was carried out using the ABI/Inform, EBSCO, Google Scholar, and Science Direct databases for the study. The review of 2071 articles yielded 37 articles for the study. The Delphi and/or AHP were the most applied tools found in the review. Finally, the study examined 18 articles that applied Delphi and/or AHP methods. The review findings contribute to the literature by providing academics and practitioners with an understanding of the appropriateness of the Delphi-based AHP research framework for analyzing challenges to procurement and supply chain management in public-sector agriculture projects. Following that, a novel best-practice research framework based on the Delphi–AHP method is presented.

1. Introduction

Procurement and supply chain management challenges of public sector agricultural development projects are crucial (not only for the project’s success but also for sustainable agricultural development) [1,2,3]. In general, projects can be categorized into three categories: natural resource issues, market issues, and policy issues. The purpose of the project is to boost agricultural productivity and return on investment [4]. Furthermore, the projects make substantial contributions to effective agricultural development, which is crucial, not just for economic development but also for food security, the environment, and agricultural sustainability [5]. Effective agricultural development is a key part of maintaining sustainable supply chain management of agricultural commodities, which would help reduce system waste [2,3].

Project procurement and supply chain challenges in public sector projects in developing countries face different types of challenges, which are specific to the type of project and the country’s approach to procurement [2,6,7]. Developing countries have distinct institutional environments in their natural, political, and/or social spheres; as a result, projects become more complex across dimensions, such as communication management, stakeholder management, and procurement management [8,9]. In addition, the faulty procurement process can significantly delay projects and increase project costs [7,10]. Accordingly, public sector projects, as well as international development projects in developing countries, are more complex in comparison to developed countries [8,11,12,13]. For instance, the procurement management of public sector projects in Bangladesh is confronted with significant challenges. It is reported that Bangladesh incurs a non-trivial economic loss due to its inefficient procurement and misappropriation of funds [7]. More specifically, public procurement is impacted by factors, such as a wide range of corruption, political control, and pressure from different types of trade unions [14]. Subsequently, project costs are often substantially increased. Historically, Bangladesh has focused on implementing public-sector development projects in agriculture. More recently, the government significantly increased its annual development program (ADP) budget for the agriculture sector in its seventh five-year plan [15]. Moreover, the government has taken a number of initiatives to improve its overall public sector procurement through one of the sustainable development goal (SDG) action plans, which includes the mid-term and long-term development plans to attain SDG 12.7: ‘Promote public procurement practices that are sustainable in accordance with national policies and priorities’ [16]. Furthermore, according to the Ministry of Agriculture [17], the country wants to improve the implementation capacity of the public sector agriculture projects, which in turn will help to attain SDG 2: ‘End hunger, achieve food security and improve nutrition and promote sustainable agriculture’.

Against this backdrop, it is considered urgent and relevant to research the identification of procurement and supply chain challenges of public-sector agriculture projects to deliver more successful public-sector agriculture projects. In the current context, these challenges are not well understood and there is limited research on procurement issues in developing countries, particularly in the Bangladesh setting. Thus, an effective and applicable research methodology will assist in identifying robust research and the best approach moving forward. The identification and categorization of procurement and supply chain challenges in other industry domains were investigated using Delphi and AHP methods [18,19,20]. However, given that developing countries have some unique characteristics compared to developed countries, it would be justified to thoroughly examine the pros and cons of the methodology before researchers choose one to conduct such research. Thus, the purpose of this article was to critically study the usefulness and application of the Delphi and/or AHP method in analyzing the challenges of procurement and supply chain management in public-sector agriculture projects in Bangladesh. This will enable researchers to identify the strengths and weaknesses of the methodology before they adapt it as the solution methodology. In addition, the study recommends a best-practice research framework using the Delphi–AHP method to identify and categorize the challenges. Furthermore, the study adds to the literature by providing an understanding of an appropriate MCDM framework for both academics and practitioners in analyzing procurement and supply chain issues of public-sector development projects in a developing country context. Moreover, a novel best-practice research framework, underpinned by the Delphi–AHP method, is subsequently offered.

This paper is divided into five sections. Following the introduction, Section 2 provides a literature review to highlight the research’s contextual background. Section 3 then briefly summarizes the key methods examined in this study before outlining the systematic review method ultimately used. Following that, in Section 4, the study presents the findings of the systematic review, as well as a discussion of the methodology and their best practice approach to identifying and categorizing procurement and supply chain management challenges in public sector agriculture projects in Bangladesh. This section also emphasizes the findings’ contributions. Section 5 presents the conclusion.

2. Literature Review

Based on the literature review, this section highlights the basic understanding of the key issues and background of the study using the framework outlined below (Figure 1).

2.1. Project Procurement Management

The term “procurement” is applied in broad aspects and it is used to describe a number of entities (i.e., functions, organizations, processes, systems, and so on). According to [21], “procurement was once descriptive of the simple clerical activities associated with purchasing well-specified items, but it has evolved in some organizations to describe instead strategic partnering efforts made by senior executives”. He also argues that it is the central part of supply chain management since it integrates the processes and activities of suppliers, vendors, producers, customers, and organizations that are routinely required to shape enterprise strategy based on opportunities to form alliances, partnerships, and joint ventures with vendors. Similarly, Raymond (2008) recommends five key principles of procurement: value for money, ethics, competition, transparency, and accountability [22]. Additionally, Walker and Rowlinson [23] suggest some other issues (i.e., culture, leadership, management, environmental, economic, ethical, and political issues) that have an impact on procurement. Whereas, project procurement management is the process of obtaining products, services, or results that the project team needs from outside the project team [24] Additionally, project procurement can be examined from different “lenses”—functional, organizational, system, and process lenses [21]. A functional lens refers to and depicts specific job tasks (i.e., obtaining vendor quotations), a division of labor (i.e., buying items vs. making them), and worker skills (i.e., negotiation). Similarly, the organizational lens deals with specific departments or other organizational entities in the enterprise; managerial hierarchy; worker roles; organizational responsibilities; and so on. From a system lens view, it refers to a system, which is related to inputs (i.e., requirements, information), outputs (i.e., purchase orders, received vendor items), transfer functions (i.e., vendor management), and the environment (i.e., industry). In the case of the process lens concept, it describes a set of interlinked processes among the vendors, producers, and customers.

2.2. Project Supply Chain Management

Handfield and Nichols [25] define a supply chain as “the supply chain encompasses all organizations and activities associated with the flow and transformation of goods from the raw materials stage, through to the end-user, as well as the associated information flows” [25]. According to Chartered Institute of Procurement and Supply (2019) “a supply chain involves a network of individuals, organizations, technology, activities and resources to make sure goods or services flow along the chain” [26]. Thus, all parts of the supply chain need to work together. The basic objective of the chain is to procure and supply something to the customer or end-user, thus the chain has two stages: upstream and downstream. The Council of Supply Chain Professionals (CSCMP) defines supply chain management as encompassing the planning and management of all activities involved in sourcing and procurement, conversion, and all logistics management activities. Importantly, it also includes coordination and collaboration with channel partners, which can be suppliers, intermediaries, third-party service providers, and customers. In essence, supply chain management integrates supply and demand management within and across companies [27].

On the other hand, project supply chain management is a collection of methods used to finish and deliver a product, service, or project in such a way that the system-wide costs are minimized while still surpassing or maintaining the customers’ service level expectations [2]. Basu and Wright [28] studied total supply chain management of the projects, and the study states that six components are for supply chain configuration (e.g., customer focus and demand management, resource and capacity management, procurement and supplier focus, inventory management, operations management, and distribution management), and three components are for supply chain integration (e.g., systems and procedures, sales and operations planning, and performance management). These building blocks will be applicable, to a varying degree, to all types and strategies of supply chains, whether they are primarily pulling or pushing processes, whether they are agile or lean supply chains, or whether they are in the manufacturing or service sector. As a result of the unique nature of the activities involved in each project’s supply chain, professionals in this field must constantly adapt their strategies in order to meet the ever-changing demands of supply chain management [2].

2.3. Challenges to Public Sector Development Projects in Bangladesh

Being a developing country, Bangladesh faces several challenges in its public sector projects [29]. Consequently, the World Bank reports that Bangladesh is experiencing significant economic losses due to its inefficient procurement and misappropriation of funds [30]. Moreover, procurement delays lead to increased project costs [7]. Likewise, some other factors also affect public procurement in Bangladesh, such as extensive corruption, political control, and pressure from different types of trade unions. Therefore, procurement management should be more accountable to ensure good governance in public procurement [14]. Under the annual development program (ADP) of the country, all the public sector development projects are designed, implemented, monitored, and evaluated by the guidelines of the Ministry of Planning, the Planning commission, respective ministries, and implementing agencies [31]. Basically, the project fund comes from two sources: The Government of Bangladesh (GoB), and international development partners. Currently, the government is implementing 168 agriculture sector projects in FY 2020-21 with their specific budgets, targets, and durations. As per the project completion report (FY 2016-17), almost 80% of the projects went over budget and time due to some common issues, including: delayed start of the projects; delayed deployment of the key staff; improper procurement planning; delay in land acquisition; natural disaster, improper budgeting; wrong estimation and forecasting; improper supply chain management in some cases; and project revision [32]. Likewise, it is evident that public sector development projects might be affected by different categories of procurement and supply chain challenges [2,7,31].

2.4. Sustainable Development: Public Sector Project Procurement and Supply Chain Management

In the context of sustainable development, project procurement and supply chain management practices are critical not only to improve the operation and overall supply chain efficiency but also to focus on environmental, economic, and social issues [1,2,28,33]. In particular, development projects are implemented for the sustainable development of a country. Project procurement and supply chain management are crucial not only for ensuing values within the project but also for project deliverables [2]. Thus, the project procurement and supply chain issues of the development projects are directly and indirectly correlated with the sustainable development of the socio-economic and environmental conditions.

There is a dearth of research on procurement and supply chain issues in public-sector agriculture projects in developing countries. Consequently, selecting an appropriate methodology is a crucial prerequisite. Because they have already been utilized to analyze challenges in other sectors of Bangladesh, Delphi and AHP are deemed useful for conducting studies of this nature. This study will investigate the methods and develop a research framework using secondary data to examine the challenges associated with public-sector agriculture projects in a developing country, such as Bangladesh.

3. Methodology and Data Collection

3.1. MCDM: Delphi and AHP Techniques

The Delphi and AHP techniques are two widely adopted MCDM tools [18,19,20,34,35,36]. Thus, the techniques have been applied by academicians, professional experts, or decision-makers in order to make the best use of the information available, either explicitly or implicitly. The Delphi technique was developed by Dalkey and Helmer at the Rand Corporation in the 1950s and is an interactive methodology that is used to elicit expert opinions [37]. It has three basic unique characteristics: anonymous responses, iteration, and controlled feedback, and statistical group response. The AHP was developed by Saaty, from the University of Pittsburgh, in the 1970s [38,39]. The structured technique is used for organizing and analyzing complex decisions and uses logical and stratified steps to aggregate complex questions [38]. It deals with the formulation of MCDM problems and can be applied to examine both tangible and intangible factors [18,40]. The decision problems are transformed into a hierarchical structure having multiple levels: goals, criteria, and sub-criteria [41,42,43]. A pair-wise comparison is made at each level to make the decisions based on the knowledge and experience of the selected experts, and the process measures the priorities of multiple alternatives under a variety of valuation criteria [35,44].

3.2. Systematic Review

Systematic reviews help to build a reliable knowledge base by accumulating knowledge from several studies. Furthermore, the process minimizes biases and replicability, thus ensuring a rigorous understanding of the existing literature [45]. Accordingly, in this case, a systematic review of the application of Delphi and/or AHP methods in analyzing the procurement and supply chain-related issues has been conducted. Here, we follow the steps proposed by [46,47] and these are depicted in Figure 2. Our approach goes beyond the systematic review and assesses the relative strengths and limitations of existing methods.

3.2.1. Stage 1: Review Planning as Highlighted Earlier, the Aim of This Study Is to Review the Literature on the Delphi–AHP Method to Address the following Two Questions

Research Question 1: What is the best-practice application of the Delphi–AHP method to identify and categorize the challenges to procurement and supply chain management in the public sector agriculture projects in Bangladesh?

Research Question 2: What research framework using the Delphi–AHP method is recommended to identify and categorize the challenges to procurement and supply chain management in the public sector agriculture projects in Bangladesh?

As the review protocol is absolutely critical for a robust review, a protocol (Appendix A) was developed to proceed into the next stages of the systematic review, namely conducting the review, and reporting and dissemination [45,48,49,50].

3.2.2. Stage 2: Conducting the Review

According to the protocol (Appendix A), the ABI/Inform, EBSCO, and Science Direct databases were chosen as the sources of data for the study [2,45,51]. Furthermore, the Google Scholar database was selected as an additional source of data [52]. The review included studies that were conducted from 2000 to 2019. It also adopted four selection criteria (Appendix A) and used four search strings to select the papers [45,53]. Table 1 lists the search strings employed in this case.

In total, 2071 papers were initially identified from the primary search. In the advanced search option of the EBSCO database, ‘the source type’, ‘subject’, and ‘subject (Thesaurus terms)’ were selected as ‘academic journal’, ‘AHP’, and ‘supply chain management’, respectively, for both search strings 1 and 2 to minimize unnecessary search results. Whereas, in the advanced search option of the Google database, the ‘exact phrase’ and ‘at least one of the words’ search options were customized using ‘challenges’, and ‘procurement, supply chain, AHP, Delphi’ as keywords, respectively, for the search string 1. Similarly, the options were customized using ‘developing countries*’, and “Delphi method*” and “AHP method*”, keywords, respectively, for search string 2. In the case of search string 3, ‘public project*’ was also included as a keyword. Finally, in search string 4, the additional keyword ‘agriculture’ was included. Table 1, and more specifically Table 2a, summarize the primary search for each search string.

In the first screening, using the first and second selection criteria outlined in the protocol (refer to Appendix A), 101 papers were selected from 2071. Table 2b summarizes the results of the first selection process by each search string.

Subsequently, in the second screening, the 101 papers were further examined using the third and fourth selection criteria, which reduced our focus to 37 journal articles. Table 2c provides an outline of the results by the search string. A summary of the overall systematic review process is illustrated in Figure 3.

In the case of data analysis, it is reported that content analysis is an effective tool to review the literature in a transparent and systematic way, which means it is a relevant tool to conduct research in supply chain management and related disciplines, such as logistics or operations [51]. Thus, content analysis was employed to analyze the 37 papers identified in the selection process discussed above, and the purpose of the analysis was to address the research questions.

3.2.3. Stage 3: Reporting and Dissemination

In the final stage, we report (refer to Section 4) on the studies that apply the Delphi and/or AHP method in analyzing challenges in procurement and supply chain management. We also highlight the limitations and opportunities for further research.

4. Result and Discussion

4.1. Key Findings

The review finds that the Delphi and/or AHP methods were used by many of the studies (i.e., 18 papers out of 37) in analyzing the challenges or risks to procurement and supply chain management. Likewise, [20,36] identify the Delphi–AHP method as the solution methodology to analyze the risks in the Bangladesh pharmaceutical industry and the barriers to big data analytics in manufacturing supply chains in Bangladesh, respectively. The remaining papers used other methods and or combinations of different types of fuzzy methods such as fuzzy Delphi and AHP, fuzzy Delphi and/or fuzzy AHP, fuzzy AHP, AHP and fuzzy AHP, AHP and TOPSIS, fuzzy AHP and fuzzy TOPSIS, AHP and SAW, AHP-integrated QFD, Hybrid AHP + Fuzzy TOPSIS/PROMETHEE/Diagraph and Matrix, AHP+ DEMATEL, SCOR-FAHP, OM-AHP. Figure 4 and Appendix B provide an overview of the methods identified in the review and the corresponding references. Table 3 provides a synthesis of the studies that investigate the procurement challenges.

4.2. Relative Strengths and Limitations of the Delphi and/or AHP Method

Generally, the Delphi method is applicable where there is no specific resolution to a given policy issue and the purpose is to explore the issue with relevant experts [18]. It is argued that Fuzzy Delphi can be applied instead of the traditional Delphi approach because the Fuzzy Delphi method has the capability to capture vagueness in data [68]. In contrast, it is argued that there is a lack of statistical evidence to support this claim [85].

The AHP method is a useful tool for decision-makers to assist with prioritizing and ranking alternatives using a pairwise comparison [20]. There are alternative methodologies beyond the AHP method to prioritize key factors in decision-making. These are TOPSIS, data envelopment analysis (DEA), Grey relational analysis, ANP, TOPSIS, VIKOR, ELECTRE, etc. Nevertheless, the AHP is superior to those methods in terms of achieving a clear understanding of the relative importance of various factors, along with the ability to check for inconsistencies and its simplicity in use [55,86]. Alternatively, it is also argued that the AHP gives unbalanced results due to the unequal scale of judgments from the expert panel. Thus, several studies offer several extensions of the AHP method [20], such as the fuzzy-AHP, Pythagorean fuzzy AHP, and fuzzy inference system. However, it is argued that those fuzzy approaches do not yield more valid results compared to AHP [87]. Similarly, it is claimed that the MCDM technique, the Best Worst Method (BWM), performs better in acquiring more consistent results compared to the analytical results of the AHP method [88]. Moreover, it is also claimed that the BWM requires fewer comparisons of the alternatives compared to other MCDMs, such as AHP, DEMATEL, fuzzy AHP, Grey-DEMATEL, etc. The BWM requires a comparison between the best and all other criteria, and between others and the worst criterion [88,89,90].

Notably, it is difficult to recruit the respondents required to conduct similar types of research in developing countries due to limited expertise [7]. Similarly, being a developing country, Bangladesh also has weaknesses in its organizational and staff capacity, information management, procurement practices, effective procurement, and accountability [29]. Thus, the AHP can be applied in such cases since the method does not require a large number of samples [7]. In contrast, the studies used a combination approach of the Delphi technique and the AHP method (Delphi–AHP) as a solution to analyze supply chain barriers/risks in the manufacturing and pharmaceutical industry in Bangladesh [20,36]. The Delphi–AHP method is a systematic technique that acts as a decision-making tool and offers a logical means to identify and categorize risks or challenges. Moreover, because the Delphi–AHP method applies the purposive and snowball sampling techniques to identify the required respondents or participants, it does not require a large sample size of respondents [7,19,91,92]. Consequently, Moktadir et al. [20,93] recommend the Delphi–AHP method for analyzing the challenges or risks in supply chain management.

Accordingly, the Delphi–AHP method can be applied to identify and categorize the challenges of procurement and supply chain management in public sector agriculture projects in Bangladesh. Ahsan and Paul [7] applied AHP in analyzing procurement challenges to donor-funded public sector projects in Bangladesh. However, the challenges were listed from the literature on other industry domains. As the public sector development projects in developing countries are complex in nature and face some idiosyncratic challenges [8,94], the challenges found in other industry domains might be less significant than those in a Bangladesh setting. The application of the Delphi method appears to be a useful way to identify the relevant and significant challenges to the specific domain. The modified Delphi method—the inclusion of a literature review to identify the primary challenge before conducting the interview—could also be an effective tool in this regard [95]. Consequently, a combination of the Delphi and AHP techniques appears to be useful in exploring the issues.

4.3. Best-Practice Approach: Best-Practice Framework using Delphi–AHP Method

Based on the systematic review findings presented in the previous section, a five-step best-practice research framework that draws on the Delphi–AHP method was developed. More specifically, the framework offers a novel approach to identify and analyze the challenges to procurement and supply chain management in public sector agriculture projects in a developing country context, including Bangladesh. Figure 5 depicts the novel best-practice research framework. The following provides a sequential discussion of the key phases in the framework.

4.3.1. Step I: Literature Review to Determine the Primary List of Challenges

In the first instance, a primary list of the challenges needs to be produced from the literature review, preferably drawing on a content analysis [20,36,51,82,83].

4.3.2. Step II: Selection of Most Relevant Challenges through the Delphi Technique Using Semi-Structured Questionnaire and Experts’ Opinion

The standard Delphi method involves five steps: (1) selection of experts; (2) performing the first-round survey; (3) performing the second-round survey; (4) performing the third-round survey; and (5) synthesizing expert opinions to reach a consensus [19]. However, steps (3) and (4) are normally repeated until the process finds a final consensus [19,34]. Notably, it is argued that the selection of the Delphi design depends on the situation directed by the research problem rather than the method requirements [96]; therefore, the study recommends ten types of Delphi designs: classical, modified, decision, policy, real-time, e-Delphi, technological, online, argument, and disaggregated policy [97].

The modified Delphi method can be customized according to the requirements of the research questions [95]. The method can simplify the second step by adopting a literature review and expert interview instead of the conventionally adopted survey mentioned in the second step. The modified method uses a structured questionnaire, which has been developed with reference to the literature [19]. It is also argued that if the questionnaire is developed from the literature, the first-round survey can be omitted or skipped from the process [91,98,99].

Since there is not much research on project procurement and supply chain challenges in the Bangladesh agricultural sector project context, a semi-structured questionnaire that includes the primary challenge list found in the literature review could be useful for the first round of the Delphi technique [19,100]. The questionnaire would permit the experts to select the challenges based on their practical experience along with the primary challenge list. In addition, experts are able to include additional challenges within the list [100]. Any given study can continue to conduct additional rounds until a consensus is achieved [34].

It is evident that the experts need to be selected based on their knowledge of the issues under investigation to ensure accurate representation [101,102]. In the current situation, we suggest that for the Delphi–AHP analysis, a group of 10 to 30 anonymous experts from the Bangladesh public sector agriculture projects be chosen based on their position, work experience, current role in procurement management, and background knowledge. This is done using purposeful sampling and snowball techniques [7,19,20,36,91,103]. Then, we suggest that the semi-structured questionnaire and interview protocol be used to obtain the opinions of experts through several rounds of personal interviews, phone calls, and e-mails to choose the most important challenges [20,36,104].

4.3.3. Step III: Development of the Framework having Major and Sub-Categories of Challenges for AHP Analysis

After identifying the most relevant challenges, a framework (hierarchical structure) having major and sub-categories of challenges needs to be developed through the Delphi technique using a semi-structured questionnaire and expert opinions so that it can be used in the AHP analysis [7,20,36,105]. The framework could have three levels: challenges of procurement and supply chain in Bangladesh public sector agriculture projects (level 1), major challenges (level 2), and sub-challenges, as shown in Figure 6.

4.3.4. Step IV: Prioritizing the Selected Challenges Using the AHP Approach

At this stage, two types of pairwise comparison matrices should be prepared using Saaty’s scale [106]: among the major challenges and sub-challenges. The scale (Table 4) applies to the pairwise comparisons in AHP is a one-to-nine scale. For a set of n criteria in a matrix, (n2 − n)/2 judgments are needed, and the remaining judgments are reciprocals [7,87].

The AHP method is used to estimate the relative importance of decision-making through stratification and class analysis, and the method follows several steps [35,44]. The most important step in the AHP application is the determination of classes of decision-making. In the second step, evaluation data are collected using a pairwise comparison between decision-making factors. It also draws the matrix through the comparison in a sub-class that enables the goal of each factor to be accomplished. The data are collected through the AHP questionnaire, which has Saaty’s scale. In the third step, the weights and relative importance of the determining factors within each class are estimated using the matrix of pairwise comparison. The W₁, W₂, .....W_n are calculated, indicating the effects and preference of valuation standards C₁, C₂,....C_n using the a_ij value acquired during the pairwise comparison [35]. The eigenvalue method evaluates the weight as shown in the equation [44]: A′∙W′= λ_max∙W′. Where A is the square matrix resulting from pairwise comparison, W is the eigenvector, and λ_max is the eigenvalue.

After establishing the ranking, the consistency ratios (CR = CI/RI) of the judgments need to be measured. A CR ≤ 0.1 would be recommended as acceptable; while, if CR > 0.1, the decision-makers would be asked to re-evaluate their judgments [7,107,108,109]. The consistency measurement is developed using the following two characteristics: matrix A has greater consistency as λ_max moves closer to n, and the λ_max value is always bigger than or the same as n, as shown in the following equation [35]:

Consistency Index (CI) = (λ_max − n)/(n − 1), and Consistency Ration (CR) = CI/RI

where RI means random index.

Finally, after establishing the ranking of major challenges along with their sub-challenges, these will be sent to the expert panel (established in the earlier phases) for their final validation and suggestions. After their acceptance, the sensitivity analysis needs to be conducted [20,93].

4.3.5. Step V: Sensitivity Analysis of the Rank Order of the Challenges

The sensitivity analysis is performed by changing the weights of the criteria that could affect the AHP results, and it is done to understand the rationale behind the obtained result. The final decision is made based on the synthesis of the AHP result and the sensitivity analysis [110]. Similarly, it is argued that the analysis is used to investigate whether small variations in the priority (relative importance) weight assigned to the major or sub-categories of attribute changes (increase or decrease) affect the priority ranking of the attributes [7,42,111]. It helps to establish whether the identified attributes (i.e., the challenges) are stable. However, several weighting schemas can be applied to conduct the analysis [112]. Whereas, it is reported that the expert choice software performs five types of sensitivity analysis: performance analysis, dynamic sensitivity, gradient sensitivity, head-to-head sensitivity, and two-dimensional sensitivity [113]. The analysis can also be conducted by changing (a 10% increase or decrease) the priority weights [7]. In contrast, [55] reported that two scenarios (equal weightage and reverse weights) could be considered while conducting the analysis. Similarly, several studies examined increased priority weight values ranging from 0.1 to 0.9 [42].

4.4. Contribution of the Research Findings

In the context of examining the challenges of procurement and supply chain management, there is a dearth of literature that emphasizes the analytical technique, particularly for investigating public sector projects in developing nations. The analysis could be carried out using the Delphi and/or AHP methods. Because these tools have not yet been used to conduct these types of studies, the review study is critical in not only representing the pros and cons of the methods but also recommending a best-practice research framework based on the Delphi–AHP method. The study recommends these methods as one of the most useful tools for analyzing project supply chain issues, even though they have been used to analyze many other types of supply chain issues besides project supply chain issues. Furthermore, it is reported that there is a lack of experts on these issues in Bangladesh to be deployed as the research respondents [2,7]. So, in this situation, these tools are more useful because they do not need a large sample size to produce statistically significant results [7].

This study’s findings have a number of implications for academics, practitioners, and aspiring researchers who are involved in project supply chain management in public-sector agriculture development projects in developing countries, which are further discussed below.

• Because no substantial research has been found on the application of the Delphi and AHP method in analyzing supply chain challenges issues of public sector development projects in a developing country context, the review findings would directly contribute to the existing literature on the methodology and supply chain context. Furthermore, the study recommended a best-practice research framework along with a detailed process description based on the existing literature review. Additionally, the study also reveals some disadvantages of these tools, and it also proposes some other alternative tools (i.e., Delphi and BWM instead of Delph and AHP). Consequently, the findings would also be very useful in future research decision-making when selecting analytical tools for such research.

• Professionals involved in the planning and management of similar types of projects would greatly benefit from the methodological framework in analyzing the challenges of development project procurement and supply chain issues in a developing country context. Furthermore, because developing countries have limited resources to implement development projects, this analytical tool would be invaluable in practitioners’ decision-making processes to address the most pressing challenges more effectively while ensuring value for money.

• Furthermore, the finding has some indirect impact on society as well. One of the key objectives of the review is to identify whether the Delph and AHP methods would be applicable for analyzing procurement and supply chain challenges of public sector agriculture projects or not. According to the finding, the tools have been identified as one of the most effective tools for conducting such research activities. Such specific review findings will assist not only researchers or academics but also professionals in identifying key challenges to public sector development projects initiated for the countries’ sustainable agricultural development. As a result, the discovery will have an indirect impact on agricultural development, which will benefit social development as well.

• Finally, policymakers involved in making policies for project supply chain management approaches and challenges mitigating approaches can also benefit from these analytical tools in order to better understand the critical challenges to the project procurement and supply chain and, thus, carry out more successful agricultural development projects.

5. Conclusions

Notwithstanding the progress in the literature surrounding procurement and supply chain management challenges, several limitations warrant further investigation. For instance, there is limited literature that focuses on the analytical method in examining the challenges of procurement and supply chain management, particularly in the case of the public sector in developing countries. This study contributes to this gap by systematically reviewing the procurement and supply chain management literature and offering a best-practice framework for decision-makers specifically in the context of Bangladesh.

The Delphi and/or AHP are the most used MCDM techniques in analyzing procurement and supply chain challenges or risks—as found from the systematic review of the literature published over a nineteen-year period commencing in 2000. Several other combinations of the MCDM techniques are also notable, e.g., fuzzy Delphi and AHP, fuzzy Delphi and/or fuzzy AHP, fuzzy AHP, AHP and fuzzy AHP, AHP and TOPSIS, fuzzy AHP and fuzzy TOPSIS, AHP and SAW, AHP-integrated QFD, Hybrid AHP + Fuzzy TOPSIS/PROMETHEE/Diagraph and Matrix, AHP+ DEMATEL, SCOR-FAHP, OM-AHP.

The Delphi and/or AHP can also be useful tools in analyzing the challenges of procurement and supply chain management in public sector agriculture projects in the developing country context. Furthermore, drawing on a critical assessment of the strengths and weaknesses of the Delphi–AHP method, the study argues in favor of a combination of Delphi and AHP (Delphi–AHP) as a useful tool for identifying and categorizing the challenges or risks to procurement and supply chain management in public sector agriculture projects in Bangladesh.

Another alternative to the Delphi–AHP method can be the Delphi–BWM combination; this warrants further investigation as it has the potential to provide more consistent output within a short duration of time. This study was limited to the use of ABI/Inform, EBSCO, ScienceDirect, and Google Scholar (GS) databases. Other databases may be included in future research to expand the scope and validity of the conclusions.

Institutional Review Board Statement

The authors declare that this article does not contain any studies and/or experiments with animals performed by any of the authors.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare that they have no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures and Tables

Figure 1. Conceptual framework of the literature review.

Figure 2. Stages for conducting a systematic review [45,46].

Figure 3. Systematic review methodology flow chart.

Figure 4. Types of methods and their reference number found in the review.

Figure 5. Best-practice research framework using Delphi–AHP method (revised and adapted from the original work of [20,36].

Figure 6. Hierarchical structure of challenges.

Table 1

Selection of literature review papers.

Search Strings		Primary Selection	First Selection	Second Selection
1	(Procurement OR purchase) AND (“supply chain*”) AND (challenge OR barrier) AND (“Delphi method*” OR “AHP method*”)	1141	52	13
2	(“Delphi method*” OR “AHP method*”) AND (procurement* OR “supply chain*”) AND (challenge* OR factor*) AND (“developing countries*”)	560	16	5
3	(“Delphi method*” OR “AHP method*”) AND (procurement* OR “supply chain*”) AND (challenge* OR factor*) AND (“public project*”) AND (“developing countries*”)	225	20	5
4	(“Delphi method*” OR “AHP method*”) AND (procurement* OR “supply chain*”) AND (challenge* OR factor*) AND (“public project*”) AND (agriculture*) AND (“developing countries*”)	145	13	6

Source: Compiled by the Authors.

Table 2

(a) Primary Search. (b) First Selection. (c) Second selection.

(a) Primary Search
String (ST)	Primary Search
	ABI	GS	EBSCO	Science Direct
ST-1	420	369	3	349
ST-2	327	10	2	221
ST-3	5	21	195	4
ST-4	1	7	136	01
Total:	Primary search = 2071
(b) First Selection
String (ST)	First Selection		String (ST)	First Selection
	ABI			ABI
ST-1	16		12	0
ST-2	9		1	1
ST-3	1		2	17
ST-4	0		2	11
Total:	First selection = 101
(c) Second Selection
String (ST)	Second Selection		String (ST)	Second Selection
	ABI			ABI
ST-1	6		7	0
ST-2	4		0	1
ST-3	0		0	5
ST-4	0		2	4
Total:	Second selection = 37

Source: Compiled by the Authors.

Table 3

Studies analyzing challenges to procurement and supply chain: selected dimensions.

References	Challenges/Factors/Risks	Procurement/Supply Chain	Industry/Sector/Project	Study Country
[54]	Challenges	Green procurement	Public healthcare	Australia
[7]	Challenges	Procurement	ID public projects	Bangladesh
[55]	Factors	Procurement	Automotive and logistic	Not specified
[18]	Barriers	Reverse logistics	Electrical–electronic	Brazil
[56]	Barriers	Supply chain	Apparel	Bangladesh
[57]	Risks	Supply chain	Telecom	China
[58]	Risks	Supply chain	Healthcare	Nigeria
[59]	Risks	Inbound supply chain management	Not specified	Not specified
[60]	Risks	Supply chain	Not specified	Not specified
[61]	Barriers	Food supply chain	Dairy Food	Greece
[42]	Barrier	Green supply chain	Manufacturing	India
[62]	Risks	Supply chain	Automotive	Iran
[63]	Risks	Supply chain	Pharmaceutical	Iran
[64]	Driver, enabler, and resistor	Logistics	Agri-logistic	India
[65]	Risks	Supply chain	Coastal port	China
[66]	Barriers	Supply chain management	Humanitarian	India
[67]	Factors	Supply chain	Global supply chain information sharing	Not specified
[68]	Barriers	Supply chain	Sugar manufacturing	India
[69]	Barriers	Supply chain	Plastic manufacturing	India
[70]	Risks	Supply chain	Electronic	Taiwan
[71]	Barrier	Supply chain	Manufacturing	Not specified
[72]	Risks	Green supply chain	Pharmaceutical	India
[73]	Risks	Green supply chain	Plastic engineering	India
[69]	Success factors	Reverse logistics	Manufacturing	India
[74]	Barrier	Supply chain	Auto manufacture	India
[20]	Risks	Supply chain	Pharmaceutical	Bangladesh
[36]	Barriers	Supply chain	Manufacturing	Bangladesh
[75]	Risks	Green supply chain	Textile industry	Pakistan
[76]	Barriers	Supply chain	Humanitarian	Iran
[77]	Risks	Supply chain	Not specified	Not specified
[78]	Risks	Supply chain	Manufacturing	US
[79]	Risks	Reverse logistics	Plastic recycling	India
[80]	Risks	Supply chain	Automotive	India
[81]	Factors	Supply chain	Not specified	India
[82]	Irregularities	Procurement	Public construction	India
[83]	Factors	Green supply chain	Construction	India
[84]	Risks	Inbound supply chain management	Not specified	Not specified

Table 4

Saaty’s 1-9 scale for AHP preference.

Importance Intensity	Definition	Explanation
1	Equal Importance	Two activities contribute equally to the objective
3	Moderate importance	Experience and judgement slightly favor one over another
5	Strong importance	Experience and judgement strongly favor one over another
7	Very strong importance	Activity is strongly favored, and its dominance is demonstrated in practice
9	Absolute importance	The importance of one over another, affirmed in the highest possible order
2, 4, 6, 8	Intermediate values	Used to represent the compromise between the priorities listed above

Appendix A

Table A1

Protocol for conducting the systematic literature review.

Objectives of the study:	Refer to Section 1 (Introduction).
Research questions:	Refer to Section 3.2.1.
Strategy to identify studies:	Development of the search strings (Table 1).
	Searching of ABI/Inform, EBSCO, Google Scholar (GS), ScienceDirect databases.
	Search on 18-year period (2000–2019).
Selection of the studies:	Selection criteria 1: titles and abstracts screening;
	Selection criteria 2: introduction, methodology, conclusion and looking over the paper’s content.
	Selection criteria 3: assessment about the quality of journals (scholarly and peer-reviewed journals), accessibility (English language papers), theoretical (Delphi or AHP method in identifying challenges to procurement and/or supply chain management in public sector agriculture project in developing countries) and empirical content (qualitative) and unit of analysis (research methodology along with the analytical tools).
	Selection criteria 4: quality appraisal of papers. Papers should be assessed according to its argument, contribution, theoretical bases and methodological rigor.
Data extraction andmonitoring process:	Reading of full papers focusing on the methodology; Identification of informational contents with respect to research questions.
Data analysis and synthesis:	Descriptive content analysis by crossing data from different concepts and literature.
	Answering research questions from what is already answered in the literature.

Source: Adapted from [45,46,50].

Appendix B

Table A2

Types of Delphi and/or AHP methods found in the review.

Methods	Reference
Literature review (LR) + Delphi	[67,82,83]
Literature review + Delphi + AHP	[20,93]
Literature review + AHP	[7,54,58,81]
Literature review + Survey + AHP	[61]
Literature review + Expert opinion + AHP	[42,71,80,84,114]
Case study + AHP	[55,60,78]
Case study + SCOR-FAHP	[65]
Literature review + Fuzzy Delphi + AHP	[18]
Literature review + Expert opinion + Fuzzy AHP	[59,64,73,74]
Literature review + Expert opinion + AHP + Fuzzy AHP	[68]
Literature review + Expert opinion + Fuzzy Delphi + Fuzzy AHP	[72]
Literature review + Expert opinion + Fuzzy Delphi + BWM	[76]
Literature review + AHP + TOPSIS	[70]
Literature review + Hybrid AHP + Fuzzy TOPSIS/PROMETHEE/Diagraph and Matrix	[79]
Case + Fuzzy AHP + Fuzzy TOPSIS	[75]
Case + Literature review + Fuzzy AHP + Fuzzy TOPSIS	[62]
Expert opinion + Fuzzy AHP + Fuzzy TOPSIS	[66]
Survey + Fuzzy AHP + Fuzzy TOPSIS	[77]
Literature review + Expert opinion + AHP + SAW	[63]
Literature review + Expert opinion + AHP+ DEMATEL	[69]
Literature review + AHP-integrated QFD approach	[56]
Literature review + OM-AHP	[57]

References

1. Jum’a, L.; Zimon, D.; Ikram, M. A relationship between supply chain practices, environmental sustainability and financial performance: Evidence from manufacturing Companies in Jordan. Sustainability; 2021; 13, 2152. [DOI: https://dx.doi.org/10.3390/su13042152]

2. Khan, M.R.; Alam, M.J.; Tabassum, N.; Burton, M.; Khan, N.A. Investigating supply chain challenges of public sector agriculture development projects in Bangladesh: An application of modified Delphi-BWM-ISM approach. PLoS ONE; 2022; 17, e0270254. [DOI: https://dx.doi.org/10.1371/journal.pone.0270254] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/35731792]

3. Sarkar, B.; Debnath, A.; Chiu, A.S.F.; Ahmed, W. Circular economy-driven two-stage supply chain management for nullifying waste. J. Clean. Prod.; 2022; 339, 130513. [DOI: https://dx.doi.org/10.1016/j.jclepro.2022.130513]

4. Winters, P.; Maffioli, A.; Salazar, L. Introduction to the special feature: Evaluating the impact of agricultural projects in developing countries. J. Agric. Econ.; 2011; 62, pp. 393-402. [DOI: https://dx.doi.org/10.1111/j.1477-9552.2011.00296.x]

5. Gil, J.D.B.; Reidsma, P.; Giller, K.; Todman, L.; Whitmore, A.; van Ittersum, M. Sustainable development goal 2: Improved targets and indicators for agriculture and food security. Ambio; 2019; 48, pp. 685-698. [DOI: https://dx.doi.org/10.1007/s13280-018-1101-4]

6. Addo-Duah, P.; Westcott, T.; Mason, J.; Booth, C.; Mahamadu, A.M. Developing capability of public sector procurement in Ghana: An assessment of the road subsector client. Constr. Res. Congr.; 2014; pp. 2053-2062. [DOI: https://dx.doi.org/10.1061/9780784413517.209]

7. Ahsan, K.; Paul, S.K. Procurement issues in donor-funded international development projects. J. Manag. Eng.; 2018; 34, 04018041. [DOI: https://dx.doi.org/10.1061/(ASCE)ME.1943-5479.0000648]

8. Gasik, S. Are public projects different than projects in other sectors? preliminary results of empirical research. Procedia. Comput. Sci.; 2016; 100, pp. 399-406. [DOI: https://dx.doi.org/10.1016/j.procs.2016.09.175]

9. Golini, R.; Kalchschmidt, M.; Landoni, P. Adoption of project management practices: The impact on international development projects of non-governmental organizations. Int. J. Proj. Manag.; 2015; 33, pp. 650-663. [DOI: https://dx.doi.org/10.1016/j.ijproman.2014.09.006]

10. Ahsan, K. Determinants of the performance of public sector development projects. Int. J. Manag.; 2012; 29, pp. 77-90.

11. Abbasi, G.Y.; Al-Mharmah, H.A. Project management practice by the public sector in a developing country. Int. J. Proj. Manag.; 2000; 18, pp. 105-109. [DOI: https://dx.doi.org/10.1016/S0263-7863(98)00074-X]

12. Ahsan, K.; Gunawan, I. Analysis of cost and schedule performance of international development projects. Int. J. Proj. Manag.; 2010; 28, pp. 68-78. [DOI: https://dx.doi.org/10.1016/j.ijproman.2009.03.005]

13. Damoah, I.S.; Akwei, C.A.; Oduro Amoako, I.; Botchie, D. Corruption as a source of government project failure in developing countries: Evidence from Ghana. Proj. Manag. J.; 2018; 49, pp. 17-33. [DOI: https://dx.doi.org/10.1177/8756972818770587]

14. Mahmood, A. Public procurement and corruption in bangladesh confronting the challenges and opportunities. J. Public Adm. Policy Res.; 2010; 2, pp. 103-111.

15. Planning Commission (PC). The 7th Five Year Plan (FY 2016-FY2020). 2015; Available online: https://www.plancomm.gov.bd/sites/default/files/files/plancomm.portal.gov.bd/files/aee61c03_3c11_4e89_9f30_d79639595c67/7th_FYP_18_02_2016.pdf (accessed on 12 December 2020).

16. Hossain, M. Sustainable Procurement Is the Agenda. 2018; Available online: Cptu.gov.bd/upload/publication/2018-08-16-15-45-40-Newsletter-CPTU-July-2018.pdf (accessed on 26 April 2020).

17. Ministry of Agriculture (MoA). Seventh Five Year Plan (7FYP). 2016; Available online: https://moa.portal.gov.bd/sites/default/files/files/moa.portal.gov.bd/policies/f2005dba_0b1c_4d25_a4dd_e2e0f3b53ee7/Seventh_FYP_MoA.pdf (accessed on 12 November 2020).

18. Bouzon, M.; Govindan, K.; Rodriguez, C.M.T.; Campos, L.M.S. Identification and analysis of reverse logistics barriers using Fuzzy Delphi Method and AHP. Resour. Conserv. Recycl.; 2016; 108, pp. 182-197. [DOI: https://dx.doi.org/10.1016/j.resconrec.2015.05.021]

19. Hsu, P.; Lin, F. Developing a decision model for brand naming using Delphi method and Analytic Hierarchy Process. Asis Pac. J. Mark. Logist.; 2013; 25, pp. 187-199. [DOI: https://dx.doi.org/10.1108/13555851311314013]

20. Moktadir, M.; Ali, S.M.; Mangla, S.K.; Sharmy, T.A.; Luthra, S.; Mishra, N.; Garza-Reyes, J.A. Decision modeling of risks in pharmaceutical supply chains. Ind. Manag. Data Syst.; 2018; 118, pp. 1388-1412. [DOI: https://dx.doi.org/10.1108/IMDS-10-2017-0465]

21. Nissen, M.E. Procurement: Process overview and emerging project management techniques. Chapter Eleven The Wiley Guide to Project Technology, Supply Chain & Procurement Management; Morris, P.; Pinto, J. John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2009.

22. Raymond, J. Benchmarking in public procurement. Benchmarking Int. J.; 2008; 15, pp. 782-793. [DOI: https://dx.doi.org/10.1108/14635770810915940]

23. Rowlinson, S.; Walker, D. Culture and its impact upon project procurement. Procurement Systems: A Cross-Industry Project Management Perspective; 1st ed. Taylor & Francis Group: London, UK, 2007; Chapter 8 pp. 301-334. [DOI: https://dx.doi.org/10.4324/9780203939697]

24. PMI. PMBOK Guide|Project Management Institute. PMBOK Guide. 2017; Available online: https://www.pmi.org/pmbok-guide-standards/foundational/PMBOK (accessed on 16 April 2022).

25. Handfield, R.; Nichols, J.E.L. Supply Chain Redesign: Transforming Supply Chains into Integrated Value Systems; Ft Press: Upper Saddle River, NJ, USA, 2002.

26. Chattered Institute of Procurement and Supply (CIPS). Scope and Influence of Procurement and Supply; The Chartered Institute of Procurement & Supply (CIPS): Stamford, UK, 2019; PE9 3NZ ISBN 978-1-86124-288-4

27. Council of Supply Chain Management Professionals (CSCMP). Supply Chain Management Terms and Glossary. 2013; Available online: https://cscmp.org/CSCMP/Educate/SCM_Definitions_and_Glossary_of_Terms.aspx (accessed on 23 September 2022).

28. Basu, R.; Wright, J.N. Managing Global Supply Chains; 2nd ed. Routledge: London, UK, 2016; pp. 1-469. [DOI: https://dx.doi.org/10.4324/9781315627410]

29. Parvez, M. Analyzing Public Procurement Process Operational Inefficiencies in Bangladesh: A Study on Department of Public Health Engineering (dphe). Ph.D. Thesis; BRAC University: Dhaka, Bangladesh, 2016.

30. World Bank (WB). An Overview of the World Bank Group’s Work in Bangladesh. 2017; Available online: https://www.worldbank.org/en/country/bangladesh (accessed on 22 October 2017).

31. Hamiduzzaman, M. Planning and managing of development projects in Bangladesh: Future challenges for government and private organizations. J. Public Adm. Policy Res.; 2014; 6, pp. 16-24. [DOI: https://dx.doi.org/10.5897/JPAPR2011.050]

32. IMED. Project Completion Report. Implementation Monitoring and Evaluation Division (IMED), Bangladesh. 2020; Available online: https://imed.portal.gov.bd/sites/default/files/files/imed.portal.gov.bd/page/315d64be_c080_478a_8b14_fddea2e03f02/MinofAgriculture_23_5_20191.docx (accessed on 29 August 2020).

33. Lysenko-Ryba, K.; Zimon, D. Customer behavioral reactions to negative experiences during the product return. Sustainability; 2021; 13, 448. [DOI: https://dx.doi.org/10.3390/su13020448]

34. Brady, S.R. Utilizing and adapting the Delphi method for use in qualitative research. Int. J. Qual. Methods; 2015; 14, 1609406915621381. [DOI: https://dx.doi.org/10.1177/1609406915621381]

35. Kim, M.; Jang, Y.C. Application of Delphi-AHP methods to select the priorities of weee for recycling in a waste management decision-making tool. J. Environ. Manag.; 2013; 128, pp. 941-948. [DOI: https://dx.doi.org/10.1016/j.jenvman.2013.06.049] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/23892135]

36. Moktadir, M.A.; Ali, S.M.; Kusi-Sarpong, S.; Shaikh, M.A.A. Assessing challenges for implementing industry 4.0: Implications for process safety and environmental protection. Process Saf. Environ. Prot.; 2018; 117, pp. 730-741. [DOI: https://dx.doi.org/10.1016/j.psep.2018.04.020]

37. Hsu, C.; Sandford, B. The Delphi technique: Making sense of consensus. Pract. Assess. Res. Eval.; 2007; 12, 10. [DOI: https://dx.doi.org/10.7275/pdz9-th90]

38. Lo, C.Y.; Huang, C.Y.; Huang, C.J. The application of modified Delphi method and AHP to analyze the influential factors of leading vegetarianism trends in Taiwan. IJRRAS; 2016; pp. 41-50.

39. Taleai, M.; Nasir, K.; Mansourian, A. Using Delphi-AHP method to survey major factors causing urban plan implementation failure. Artic. J. Appl. Sci.; 2008; 8, pp. 2746-2751. [DOI: https://dx.doi.org/10.3923/jas.2008.2746.2751]

40. Jiang, Z.; Zhang, H.; Sutherland, J.W. Development of multi-criteria decision-making model for remanufacturing technology portfolio selection. J. Clean. Prod.; 2011; 19, pp. 1939-1945. [DOI: https://dx.doi.org/10.1016/j.jclepro.2011.07.010]

41. Dey, P.K.; Cheffi, W. Green supply chain performance measurement using the analytic hierarchy process: A comparative analysis of manufacturing organisations. Prod. Plan. Control.; 2013; 24, pp. 702-720. [DOI: https://dx.doi.org/10.1080/09537287.2012.666859]

42. Govindan, K.; Kaliyan, M.; Kannan, D.; Haq, A.N. Barriers analysis for green supply chain management implementation in Indian industries using Analytic Hierarchy Process. Int. J. Prod. Econ.; 2014; 147, pp. 555-568. [DOI: https://dx.doi.org/10.1016/j.ijpe.2013.08.018]

43. Madaan, J.; Mangla, S. Decision Modeling Approach for Eco-Driven Flexible Green Supply Chain. Systemic Flexibility and Business Agility; Sushil,; Chroust, G. Springer: New Delhi, India, 2015; pp. 343-364. [DOI: https://dx.doi.org/10.1007/978-81-322-2151-7_]

44. Saaty, T. The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation; RWS Publications: Pittsburgh, PA, USA, 1996.

45. Pereira, C.R.; Christopher, M.; Lago Da Silva, A. Achieving supply chain resilience: The role of procurement. Supply Chain. Manag.; 2014; 19, pp. 626-642. [DOI: https://dx.doi.org/10.1108/SCM-09-2013-0346]

46. Tranfield, D.; Denyer, D.; Smart, P. Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br. J. Manag.; 2003; 14, pp. 207-222. [DOI: https://dx.doi.org/10.1111/1467-8551.00375]

47. Tranfield, D.; Denyer, D.; Marcos, J.; Burr, M. Co-producing management knowledge. Manag. Decis.; 2004; 42, pp. 375-386. [DOI: https://dx.doi.org/10.1108/00251740410518895]

48. Brereton, P.; Kitchenham, B.; Budgen, D.; Turner, M.; Khalil, M. Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Softw.; 2007; 80, pp. 571-583. [DOI: https://dx.doi.org/10.1016/j.jss.2006.07.009]

49. Okoli, C.; Schabram, K. A Guide to Conducting a Systematic Literature Review of Information Systems Research. Sprouts Work. Pap. Inf. Syst.; 2010; 10, pp. 1-50. [DOI: https://dx.doi.org/10.2139/ssrn.1954824]

50. Xiao, Y.; Watson, M. guidance on conducting a systematic literature review. J. Plan. Educ. Res.; 2019; 39, pp. 93-112. [DOI: https://dx.doi.org/10.1177/0739456X17723971]

51. Seuring, S.; Gold, S. Conducting content-analysis based literature reviews in supply chain management. Supply Chain. Manag.; 2012; 17, pp. 544-555. [DOI: https://dx.doi.org/10.1108/13598541211258609]

52. Haddaway, N.R.; Collins, A.M.; Coughlin, D.; Kirk, S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS ONE; 2015; 10, e0138237. [DOI: https://dx.doi.org/10.1371/journal.pone.0138237]

53. Miles, M.; Huberman, A. Qualitative Data Analysis: An Expanded Sourcebook; Sage Publications: London, UK, 1994.

54. Ahsan, K.; Rahman, S. Green public procurement implementation challenges in Australian public healthcare sector. J. Clean. Prod.; 2017; 100, pp. 181-197. [DOI: https://dx.doi.org/10.1016/j.jclepro.2017.03.055]

55. Basu, R.J.; Subramanian, N.; Gunasekaran, A.; Palaniappan, P.L.K. Influence of non-price and environmental sustainability factors on truckload procurement process. Ann. Oper. Res.; 2017; 250, pp. 363-388. [DOI: https://dx.doi.org/10.1007/s10479-016-2170-z]

56. Chowdhury, M.M.H.; Umme, N.J.; Nuruzzaman, M. Strategies for mitigating supply-side barriers in the apparel supply chain: A study on the apparel industry of Bangladesh. Glob. J. Flex. Syst. Manag.; 2018; 19, pp. 41-52. [DOI: https://dx.doi.org/10.1007/s40171-017-0180-6]

57. Dong, Q.; Cooper, O. An orders-of-magnitude AHP supply chain risk assessment framework. Int. J. Prod. Econ.; 2016; 182, pp. 144-156. [DOI: https://dx.doi.org/10.1016/j.ijpe.2016.08.021]

58. Enyinda, C.; Gebremikael, F.; Ogbuehi, A.O. An analytical model for healthcare supply chain risk management. Afr. J. Bus. Econ. Res.; 2014; 9, pp. 13-27.

59. Ganguly, K.K.; Guin, K.K. A Fuzzy AHP approach for inbound supply risk assessment. Benchmarking; 2013; 20, pp. 129-146. [DOI: https://dx.doi.org/10.1108/14635771311299524]

60. Gaudenzi, B.; Borghesi, A. Managing risks in the supply chain using the AHP method. Int. J. Logist. Manag.; 2006; 17, pp. 114-136. [DOI: https://dx.doi.org/10.1108/09574090610663464]

61. Ghadge, A.; Kaklamanou, M.; Choudhary, S.; Bourlakis, M. Implementing environmental practices within the Greek dairy supply chain drivers and barriers for SMEs. Ind. Manag. Data Syst.; 2017; 117, pp. 1995-2014. [DOI: https://dx.doi.org/10.1108/IMDS-07-2016-0270]

62. Heidari, S.S.; Khanbabaei, M.; Sabzehparvar, M. A model for supply chain risk management in the automotive industry using Fuzzy Analytic Hierarchy Process and Fuzzy TOPSIS. Benchmarking; 2018; 25, pp. 3831-3857. [DOI: https://dx.doi.org/10.1108/BIJ-11-2016-0167]

63. Jaberidoost, M.; Olfat, L.; Hosseini, A.; Kebriaeezadeh, A.; Abdollahi, M.; Alaeddini, M.; Dinarvand, R. Pharmaceutical supply chain risk assessment in Iran using Analytic Hierarchy Process (AHP) and Simple Additive Weighting (SAW) methods. J. Pharm. Policy Pract.; 2015; 8, 9. [DOI: https://dx.doi.org/10.1186/s40545-015-0029-3]

64. Jakhar, M.; Srivastava, M.K. Prioritization of drivers, enablers and resistors of agri-logistics in an emerging economy using Fuzzy AHP. Br. Food J.; 2018; 120, pp. 2166-2181. [DOI: https://dx.doi.org/10.1108/BFJ-11-2017-0608]

65. Jiang, B.; Li, J.; Shen, S. Supply chain risk assessment and control of port enterprises: Qingdao port as case study. Asian J. Shipp. Logist.; 2018; 34, pp. 198-208. [DOI: https://dx.doi.org/10.1016/j.ajsl.2018.09.003]

66. Kabra, G.; Ramesh, A. Analyzing drivers and barriers of coordination in humanitarian supply chain management under fuzzy environment. Benchmarking; 2015; 22, pp. 559-587. [DOI: https://dx.doi.org/10.1108/BIJ-05-2014-0041]

67. Kembro, J.; Näslund, D.; Olhager, J. Information sharing across multiple supply chain tiers: A Delphi study on antecedents. Int. J. Prod. Econ.; 2017; 193, pp. 77-86. [DOI: https://dx.doi.org/10.1016/j.ijpe.2017.06.032]

68. Kumar, R.; Kansara, S. Information technology barriers in Indian sugar supply chain: An AHP and Fuzzy AHP approach. Benchmarking; 2018; 25, pp. 1978-1991. [DOI: https://dx.doi.org/10.1108/BIJ-01-2017-0004]

69. Mangla, S.K.; Govindan, K.; Luthra, S. Critical success factors for reverse logistics in Indian industries: A structural model. J. Clean. Prod.; 2016; 129, pp. 608-621. [DOI: https://dx.doi.org/10.1016/j.jclepro.2016.03.124]

70. Lo, S.; Chen, J.; Liu, T. Response to demand uncertainty of supply chains: A value-focused approach with AHP and TOPSIS. Int. J. Ind. Eng.; 2018; 25, pp. 739-756.

71. Kumar, S.; Luthra, S.; Haleem, A. Benchmarking supply chains by analyzing technology transfer critical barriers using AHP approach. Benchmarking; 2015; 22, pp. 538-558. [DOI: https://dx.doi.org/10.1108/BIJ-05-2014-0040]

72. Kumar, A.; Zavadskas, E.K.; Mangla, S.K.; Agrawal, V.; Sharma, K.; Gupta, D. When risks need attention: Adoption of green supply chain initiatives in the pharmaceutical industry. Int. J. Prod. Res.; 2019; 57, pp. 3554-3576. [DOI: https://dx.doi.org/10.1080/00207543.2018.1543969]

73. Mangla, S.K.; Kumar, P.; Barua, M.K. Risk analysis in green supply chain using Fuzzy AHP approach: A case study. Resour. Conserv. Recycl.; 2015; 104, pp. 375-390. [DOI: https://dx.doi.org/10.1016/j.resconrec.2015.01.001]

74. Mangla, S.; Govindan, K.; Luthra, S. Prioritizing the barriers to achieve sustainable consumption and production trends in supply chains using Fuzzy Analytical Hierarchy Process. J. Clean. Prod.; 2017; 151, pp. 509-525. [DOI: https://dx.doi.org/10.1016/j.jclepro.2017.02.099]

75. Nazam, M.; Xu, J.; Tao, Z.; Ahmad, J.; Hashim, M. A Fuzzy AHP-TOPSIS framework for the risk assessment of green supply chain implementation in the textile industry. Int. J. Supply Oper. Manag.; 2015; 2, pp. 548-568. [DOI: https://dx.doi.org/10.22034/2015.1.02]

76. Sahebi, I.; Arab, A.; Moghadam, M. Analyzing the barriers to humanitarian supply chain management: A case study of the Tehran Red Crescent Societies. Int. J. Disaster Risk Reduct.; 2017; 24, pp. 232-241. [DOI: https://dx.doi.org/10.1016/j.ijdrr.2017.05.017]

77. Samvedi, A.; Jain, V.; Chan, F.T.S. Quantifying risks in a supply chain through integration of Fuzzy AHP and Fuzzy TOPSIS. Int. J. Prod. Res.; 2013; 51, pp. 2433-2442. [DOI: https://dx.doi.org/10.1080/00207543.2012.741330]

78. Schoenherr, T.; Tummala, V.; Harrison, T. Assessing supply chain risks with the Analytic Hierarchy Process: Providing decision support for the offshoring decision by a US manufacturing company. J. Purch. Supply Manag.; 2008; 14, pp. 100-111. [DOI: https://dx.doi.org/10.1016/j.pursup.2008.01.008]

79. Senthil, S.; Murugananthan, K.; Ramesh, A. Analysis and prioritisation of risks in a reverse logistics network using hybrid multi-criteria decision-making methods. J. Clean. Prod.; 2018; 179, pp. 716-730. [DOI: https://dx.doi.org/10.1016/j.jclepro.2017.12.095]

80. Sharma, S.; Bhat, A. Identification and assessment of supply chain risk: Development of AHP model for supply chain risk prioritisation. Int. J. Agil. Syst. Manag.; 2012; 5, pp. 350-369. [DOI: https://dx.doi.org/10.1504/IJASM.2012.050155]

81. Singh, R.K. Prioritizing the factors for coordinated supply chain using Analytic Hierarchy Process (AHP). Meas. Bus. Excell.; 2013; 17, pp. 80-97. [DOI: https://dx.doi.org/10.1108/13683041311311383]

82. Tabish, S.Z.S.; Jha, K.N. Analyses and evaluation of irregularities in public procurement in India. Constr. Manag. Econ.; 2011; 29, pp. 261-274. [DOI: https://dx.doi.org/10.1080/01446193.2010.549138]

83. Wibowo, M.A.; Handayani, N.U.; Mustikasari, A. Factors for implementing green supply chain management in the construction industry. J. Ind. Eng. Manag.; 2018; 11, pp. 651-679. [DOI: https://dx.doi.org/10.3926/jiem.2637]

84. Wu, T.; Blackhurst, J.; Chidambaram, V. A model for inbound supply risk analysis. Comput. Ind.; 2006; 57, pp. 350-365. [DOI: https://dx.doi.org/10.1016/j.compind.2005.11.001]

85. Saffie, N.A.M.; Shukor, N.A.M.; Rasmani, K.A. Fuzzy Delphi Method: Issues and challenges. Proceedings of the International Conference on Logistics, Informatics and Service Sciences, LISS 2016; Sydney, Australia, 24–27 July 2016; [DOI: https://dx.doi.org/10.1109/LISS.2016.7854490]

86. Luthra, S.; Govindan, K.; Kannan, D.; Mangla, S.K.; Garg, C.P. An integrated framework for sustainable supplier selection and evaluation in supply chains. J. Clean. Prod.; 2017; 140, pp. 1686-1698. [DOI: https://dx.doi.org/10.1016/j.jclepro.2016.09.078]

87. Saaty, T.L. There is no mathematical validity for using fuzzy number crunching in the Analytic Hierarchy Process. J. Syst. Sci. Syst. Eng.; 2006; 15, pp. 457-464. [DOI: https://dx.doi.org/10.1007/s11518-006-5021-7]

88. Rezaei, J. Best-Worst multi-criteria decision-making method. Omega; 2015; 53, pp. 49-57. [DOI: https://dx.doi.org/10.1016/j.omega.2014.11.009]

89. Sadjadi, S.J.; Karimi, M. Best-Worst multi-criteria decision-making method: A robust approach. Decis. Sci. Lett.; 2018; 7, pp. 323-340. [DOI: https://dx.doi.org/10.5267/j.dsl.2018.3.003]

90. Mi, X.; Tang, M.; Liao, H.; Shen, W.; Lev, B. The state-of-the-art survey on integrations and applications of the Best Worst Method in decision making: Why, what, what for and what’s next?. Omega; 2019; 87, pp. 205-225. [DOI: https://dx.doi.org/10.1016/j.omega.2019.01.009]

91. Ameyaw, E.; Hu, Y.; Shan, M.; Chan, A.P.C.; Le, Y. Application of Delphi Method in Construction Engineering and Management Research: A Quantitative Perspective. J. Civ. Eng. Manag.; 2016; 22, pp. 991-1000. [DOI: https://dx.doi.org/10.3846/13923730.2014.945953]

92. Saaty, T.L. How to make a decision: The Analytic Hierarchy Process. Inf. J. Appl. Anal.; 1994; 24, pp. 19-43. [DOI: https://dx.doi.org/10.1287/inte.24.6.19]

93. Moktadir, M.; Ali, S.; Paul, S.; Shukla, N. Barriers to big data analytics in manufacturing supply chains: A case study from Bangladesh. Comput. Ind. Eng.; 2019; 128, pp. 1063-1075. [DOI: https://dx.doi.org/10.1016/j.cie.2018.04.013]

94. Cusworth, J.W.; Franks, T.R. Managing Projects in Developing Countries; Routledge: London, UK, 2013; [DOI: https://dx.doi.org/10.4324/9781315845111]

95. Twose, P.; Jones, U.; Cornell, G. Minimum standards of clinical practice for physiotherapists working in critical care settings in the United Kingdom: A Modified Delphi Technique. J. Intensive Care Soc.; 2018; 20, pp. 118-131. [DOI: https://dx.doi.org/10.1177/1751143718807019] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31037104]

96. Hasson, F.; Keeney, S. Enhancing rigour in the Delphi technique research. Technol. Forecast. Soc Chang.; 2011; 78, pp. 1695-1704. [DOI: https://dx.doi.org/10.1016/j.techfore.2011.04.005]

97. Keeney, S. The Delphi Technique. The Research Process in Nursing; Gerrish, K.; Anne, L. Blackwell Publishing: London, UK, 2009.

98. Ke, Y.; Wang, S.; Chan, A.P.C.; Cheung, E. Understanding the risks in China’s PPP projects: Ranking of their probability and consequence. Eng. Constr. Archit. Manag.; 2011; 18, pp. 481-496. [DOI: https://dx.doi.org/10.1108/09699981111165176]

99. Hon, C.K.H.; Chan, A.P.C.; Yam, M.C.H. Empirical study to investigate the difficulties of implementing safety practices in the repair and maintenance sector in Hong Kong. J. Constr. Eng. Manag.; 2012; 138, pp. 877-884. [DOI: https://dx.doi.org/10.1061/(ASCE)CO.1943-7862.0000497]

100. Chan, A.P.C.; Yung, E.H.K.; Lam, P.T.I.; Tam, C.M.; Cheung, S.O. Application of Delphi method in selection of procurement systems for construction projects. Constr. Manag. Econ.; 2001; 19, pp. 699-718. [DOI: https://dx.doi.org/10.1080/01446190110066128]

101. Lee, S.; Ross, S. Sport sponsorship decision making in a global market: An approach of Analytic Hierarchy Process (AHP). Sport Bus. Manag.; 2012; 2, pp. 156-168. [DOI: https://dx.doi.org/10.1108/20426781211243999]

102. Sambasivan, M.; Fei, N. Evaluation of critical success factors of implementation of iso 14001 using Analytic Hierarchy Process (AHP): A case study from Malaysia. J. Clean. Prod.; 2008; 16, pp. 1424-1433. [DOI: https://dx.doi.org/10.1016/j.jclepro.2007.08.003]

103. Okoli, C.; Pawlowski, S.D. The Delphi method as a research tool: An example, design considerations and applications. Inf. Manag.; 2004; 42, pp. 15-29. [DOI: https://dx.doi.org/10.1016/j.im.2003.11.002]

104. Avella, J.R. Delphi panels: Research design, procedures, advantages, and challenges. Int. J. Dr. Stud.; 2016; 11, pp. 305-321. [DOI: https://dx.doi.org/10.28945/3561]

105. Cheng, E.W.; Li, H. Information Priority-Setting for Better Resource Allocation Using Analytic Hierarchy Process (AHP). Inf. Manag. Comput. Secur.; 2001; 9, pp. 61-70. [DOI: https://dx.doi.org/10.1108/09685220110388827]

106. Saaty, T.L. Rank from comparisons and from ratings in the Analytic Hierarchy/Network Processes. Eur. J. Oper. Res.; 2006; 168, pp. 557-570. [DOI: https://dx.doi.org/10.1016/j.ejor.2004.04.032]

107. Saaty, T.L. An exposition of the AHP in reply to the paper “remarks on the Analytic Hierarchy Process”. Manag. Sci.; 1990; 36, pp. 259-268. [DOI: https://dx.doi.org/10.1287/mnsc.36.3.259]

108. Saaty, T.L. What is relative measurement? the ratio scale phantom. Math. Comput. Model.; 1993; 17, pp. 1-12. [DOI: https://dx.doi.org/10.1016/0895-7177(93)90170-4]

109. Saaty, T.; Kearns, K. Analytical Planning: The Organization of Systems; Pergamon Press: Oxford, UK, 1985.

110. Mu, E.; Pereyra-Rojas, M. Understanding the Analytic Hierarchy Process. Practical Decision Making; Springer: Cham, Switzerland, 2017; pp. 7-22. [DOI: https://dx.doi.org/10.1007/978-3-319-33861-3_2]

111. Bevilacqua, M.; Braglia, M. The Analytic Hierarchy Process applied to maintenance strategy selection. Reliab. Eng. Syst. Saf.; 2000; 70, pp. 71-83. [DOI: https://dx.doi.org/10.1016/S0951-8320(00)00047-8]

112. Ananda, J.; Herath, G. The use of Analytic Hierarchy Process to incorporate stakeholder preferences into regional forest planning. For. Policy Econ.; 2003; 5, pp. 13-26. [DOI: https://dx.doi.org/10.1016/S1389-9341(02)00043-6]

113. Yadav, V.; Sharma, M.K. Multi-criteria supplier selection model using the Analytic Hierarchy Process approach. J. Model. Manag.; 2016; 11, pp. 326-354. [DOI: https://dx.doi.org/10.1108/JM2-06-2014-0052]

114. Luthra, S.; Mangla, S.; Xu, L.; Diabat, A. Using AHP to evaluate barriers in adopting sustainable consumption and production initiatives in a supply chain. Int. J. Prod. Econ.; 2016; 181, pp. 342-349. [DOI: https://dx.doi.org/10.1016/j.ijpe.2016.04.001]