Evaluating critical success factors in implementing E-learning system using multi-criteria decision-making

Abstract

Learning using the Internet or training through E-Learning is growing rapidly and is increasingly favored over the traditional methods of learning and teaching. This radical shift is directly linked to the revolution in digital computer technology. The revolution propelled by innovation in computer technology has widened the scope of E-Learning and teaching, whereby the process of exchanging information has been made simple, transparent, and effective. The E-Learning system depends on different success factors from diverse points of view such as system, support from the institution, instructor, and student. Thus, the effect of critical success factors (CSFs) on the E-Learning system must be critically analyzed to make it more effective and successful. This current paper employed the analytic hierarchy process (AHP) with group decision-making (GDM) and Fuzzy AHP (FAHP) to study the diversified factors from different dimensions of the web-based E-Learning system. The present paper quantified the CSFs along with its dimensions. Five different dimensions and 25 factors associated with the web-based E-Learning system were revealed through the literature review and were analyzed further. Furthermore, the influence of each factor was derived successfully. Knowing the impact of each E-Learning factor will help stakeholders to construct education policies, manage the E-Learning system, perform asset management, and keep pace with global changes in knowledge acquisition and management.

About the Authors:

Quadri Noorulhasan Naveed

Roles Conceptualization, Formal analysis, Investigation, Writing – original draft

* E-mail: [email protected]

Affiliation: College of Computer Science, King Khalid University, Abha, Saudi Arabia

Mohamed Rafik Noor Qureshi

Roles Conceptualization, Formal analysis, Methodology

Affiliation: College of Engineering, King Khalid University, Abha, Saudi Arabia

Nasser Tairan

Roles Data curation, Resources, Validation

Affiliation: College of Computer Science, King Khalid University, Abha, Saudi Arabia

AbdulHafeez Mohammad

Roles Methodology, Resources, Software

Affiliation: Department of Computer Science, Bahria University, Lahore, Pakistan

Asadullah Shaikh

Roles Resources, Supervision, Writing – review & editing

Affiliation: College of Computer Science and Information Systems, Najran University, Najran, Saudi Arabia

ORCID logo http://orcid.org/0000-0003-4806-6159

Alhuseen O. Alsayed

Roles Formal analysis, Visualization, Writing – review & editing

Affiliation: Deanship of Scientific Research, King Abdulaziz University, Jeddah, Saudi Arabia

ORCID logo http://orcid.org/0000-0003-3044-799X

Asadullah Shah

Roles Methodology, Project administration

Affiliation: Kulliyyah of ICT, International Islamic University Malaysia, Kuala Lumpur, Malaysia

Fahad Mazaed Alotaibi

Roles Project administration, Resources, Visualization

Affiliation: Department of Information Technology, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia

Abstract

Learning using the Internet or training through E-Learning is growing rapidly and is increasingly favored over the traditional methods of learning and teaching. This radical shift is directly linked to the revolution in digital computer technology. The revolution propelled by innovation in computer technology has widened the scope of E-Learning and teaching, whereby the process of exchanging information has been made simple, transparent, and effective. The E-Learning system depends on different success factors from diverse points of view such as system, support from the institution, instructor, and student. Thus, the effect of critical success factors (CSFs) on the E-Learning system must be critically analyzed to make it more effective and successful. This current paper employed the analytic hierarchy process (AHP) with group decision-making (GDM) and Fuzzy AHP (FAHP) to study the diversified factors from different dimensions of the web-based E-Learning system. The present paper quantified the CSFs along with its dimensions. Five different dimensions and 25 factors associated with the web-based E-Learning system were revealed through the literature review and were analyzed further. Furthermore, the influence of each factor was derived successfully. Knowing the impact of each E-Learning factor will help stakeholders to construct education policies, manage the E-Learning system, perform asset management, and keep pace with global changes in knowledge acquisition and management.

Figures

Fig 6

Table 14

Fig 1

Table 1

Table 2

Fig 2

Fig 3

Table 3

Table 4

Table 5

Table 6

Table 7

Fig 4

Table 8

Table 9

Table 10

Table 11

Table 12

Table 13

Fig 5

Fig 6

Table 14

Fig 1

Table 1

Table 2

Citation: Naveed QN, Qureshi MRN, Tairan N, Mohammad A, Shaikh A, Alsayed AO, et al. (2020) Evaluating critical success factors in implementing E-learning system using multi-criteria decision-making. PLoS ONE 15(5): e0231465. https://doi.org/10.1371/journal.pone.0231465

Editor: Baogui Xin, Shandong University of Science and Technology, CHINA

Received: December 30, 2019; Accepted: March 24, 2020; Published: May 4, 2020

Copyright: © 2020 Naveed et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper.

Funding: The authors extend their appreciation to the Deanship of Scientific Research at King Khalid University for funding this work through General Research Project under grant number R.G.P.1/138/40.

Competing interests: The authors have declared that no competing interests exist.

1. Introduction

E-Learning is a state-of-the-art methodology for learning and teaching in digital environments aimed at improving education through enhancing the teaching and learning processes [1, 2]. E-Learning systems, which are exempt from time and location limitations, offer opportunities for teaching and learning and play a vital role in promoting new teaching methods [3]. It is observed that various developed and developing countries have adopted and implemented E-Learning for Teaching-Learning. It may include complete dependency through the learning management system (LMS), blended E-Learning or Traditional blackboard Teaching-Learning with E-Learning.

E-Learning is recognized as a key factor in enhancing the performance of the education sector and its stakeholders’ satisfaction. Research has shown that E-Learning provides improved ability to share information, cost-efficacy, accessibility, and easy access through the World Wide Web (www). Not only that, E-Learning supports every student in a new learning approach to achieve a high volume of interaction and an easy learning environment with the help of technology integration in education [4, 5]. Several studies have also discussed E-Learning as a common methodology that is becoming user-friendly as it provides relaxed access and usage [6, 7]. Furthermore, E-Learning allows students to study and learn freely using cutting-edge new technology without conventional teaching approaches like direct supervision and control mechanisms [8–10]. The fast development in IT with new software and high configuration of hardware have made E-Learning more easy and useful, which then improves university learning outcomes [11]. It is therefore indisputable that E-Learning has become an indispensable tool in educational technology [12].

Despite its advantages, E-Learning’s full and successful implementation is yet to be achieved [13]. If E-Learning is successfully implemented into the education system, its many perceived benefits can be noticed. Past studies have indicated that critical success factors (CSFs) play a vital role in the successful implementation of E-Learning. Furthermore, it has also been revealed that the critical factors of different dimensions can have varying effects on the E-Learning system [14–16]. Therefore, it is imperative to explicitly investigate the assessment and prioritisation of E-Learning’s CSFs and propose a hierarchical model of E-Learning’s success factors. Such a model will allow E-Learning administrators and other stakeholders to recognise and start paying attention to the most important E-Learning success factors. The E-Learning needs a huge investment by the management to implement in the education system. It has been studied that despite huge investment the E-Learning system usage is still low [17]. Many researchers have attempted to resolve various issues like the impact of (CSFs) on the E-Learning system, issues concerning user satisfaction, and the effect of the E-Learning system on student learning [3]. To implemnet the E-Learning system effectively, it is significant to know the CSFs that plays the vital role in successful implementaion fo E-Learning. There is little work on prioritisation and ranking of the CSFs.

To address the discussed research gaps, this current study attempts to evaluate and rank the dimensions and CSFs of the E-Learning system. An analytic hierarchy process with group decision making (AHP-GDM) and Fuzzy AHP (FAHP) based multi-criteria decision-making (MCDM) is used in the assessment and prioritization of dimensions and CSFs of the E-Learning system. The present research is undertaken to accomplish three major objectives, namely:

1. To conduct a comprehensive survey of the E-Learning dimensions and CSFs through an in-depth literature review.

2. To prioritize the E-Learning dimensions and CSFs using MCDM in crisp and fuzzy environment.

3. To provide useful recommendations based on the prioritized dimensions and CSFs of E-Learning.

This paper is organized as follows. Section 2 presents the review of literature on CSF of E-Learning and MCDM based methodology in E-Learning. Section 3 discuss the framework to identify the CSFs of E-Learning, Next, Section 4 provides the AHP-GDM and FAHP research methodologies, while the application of AHP-GDM and FAHP for prioritizing E-Learning’s CSFs are presented in Section 5. Section 3 discusses the results of the prioritization, whereas Section 7 discusses the limitations of the present study and provides scope for the future research, Section 8 concludes the present study. Finally, a detailed Recommendation of Effective E-Learning Implementation for Students, Instructors and Developers is provided in the last Section 9.

2. Literature review

CSFs are commonly used for the seamless execution of various strategies and programs by business establishments. It refers to ‘the limited number of objectives areas will ensure the organization’s successful competitive performance if they are satisfactory’ [18]’ and demands the continuous attention of managers in the selected areas. CSFs encompasses the crucial factors that are critical in the areas where performance is very important to establish the success of organizations [19]. As such, there should be a strong focus on these factors to achieve success [20]. Despite the crucial importance of dimensions and CSFs for organizational success, the limited research that may help in the successful implementation of E-Learning has been undertaken [21].

Several CSFs influence effective teaching and learning methodologies in E-Learning. Many dimensions and CSFs of E-Learning have a significant effect on the teaching-learning process. Prior studies have reported on important E-Learning CSFs. For example, a study [3] focused on effective E-Learning factors in Saudi universities through a systematic review and statistical analysis of academic staff and students’ perspectives. The study concluded that CSFs are the most effective factors for the successful implementation of E-Learning. Bhuasiri [17] studied and analyzed the CSF for E-Learning in developing countries. Abdullah and Rowley [3] carried out comparative perspective study on E-Learning CSF. Hence, CSFs must be thoroughly studied. The analysis of the dimensions and CSFs will guarantee the effective implementation of the E-Learning system.

Apart from identifying the various significant CSFs of E-Learning, a detailed literature review on various MCDM-based modeling has also been carried out and is presented in this section. Several scholars have utilized MCDM to examine the E-Learning system. For instance, FAHP was applied in a study [22] to propose a framework for learning with the help of Massive Open Online Courses (MOOC). This framework can help in increasing the effectiveness of teaching. Moreover, learners can get simplified lifelong learning with maximized motivation, while simultaneously the number of dropouts can be reduced. Other than that, a recent study [23] established a model for evaluating the contents of multimedia quality in E-Learning by calculating each attribute’s priority weights using the MCDM-based Analytic Network Process (ANP).

On the other hand, FAHP has also been used in investigating the possibility of using a new PVM-VSI method considering the variants in different criteria [24]. Another study [25] used the AHP-GDM methodology to derive a strategic plan for evaluating E-Learning applications using five different approaches at different levels. The researcher proposed a quantitative evaluation method for all the E-Learning decision-makers (DMs) from higher education institutes. Next, FAHP was employed to rank various E-Learning factors to achieve E-Learning performance. The five most important factors from the students as well as the instructors’ point of view were derived [26]. Research on E-learning website selection and ranking have been carried using fuzzy Complex Proportional Assessment (COPRAS) [27], weighted distance-based approximation [28], multi attribute decision-making approaches [29], multi-attribute decision-making matrix methodology [30] and FAHP, COPRAS, VlseKriterijumska Optimizacija I Kompromisno Resenje (VIKOR), Weighted Distance Based Approximation (WDBA) [31].

The in-depth review of the literature also revealed that many researchers have combined different methodologies to get more accurate results. For example, FAHP and Step-wise Weight Assessment Ratio Analysis (SWARA) methods were applied to assess five dimensions and 24 critical E-Learning factors [32]. On the other hand, another study used AHP in the selection of software-based learning objectives (LO). SDUNESA, a web-based software, utilized the AHP method parameter for the LO selection. The implemented methodology reduced the time for searching LO in large database systems [33]. Apart from that, the research carried out by [17] applied a combined approach of MCDM to prioritize and identify E-Learning CSFs. This research found the most successful factors based on their importance level using the feedback from several stakeholders such as ICT experts and staff members. In another study [34] investigated the online learning quality using the FAHP method to study Internet resources and their environment. This study analyzed Internet learning quality using qualitative and quantitative methodologies.

Next, Kang et al. [35] used AHP to examine the different characteristics of the E-Learning system and applied the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) in ranking different criteria using derived weights. The researchers stressed that the integration of TOPSIS with AHP can be beneficial in real usage. In their study, 33 criteria for successful implementation were revealed and the AHP methodology was utilized in creating a hierarchical model and assessment of adopted E-Learning CSFs.

Based on the reviewed literature, the combination of several methods have been revealed to be much more effective and various researchers have applied it for analyzing different E-Learning features. Therefore, this current study’s focus on AHP-GDM and FAHP. The employed methods will contribute to the existing literature on to E-Learning.

3. Framework for the identification of CSFs in E-Learning

Numerous studies have established key findings concerning E-Learning. Among them, many have critically examined the dimensions and CSFs of the E-Learning system. Some studies [5], [14, 36] have classified the CSFs of E-Learning into different dimensions such as Institutional Management Service, Instructors, System and Technological, Students, and Content Design. In the first stage of this study, all possible success factors related to E-Learning were identified which resulted in 36 CSFs. Next, after critically reviewing the identified CSFs with the help of DMs, these factors were reduced to 25 and then grouped into five dimensions. The identified CSFs are further explained in this section.

The following steps were employed to find and categorize the important CSFs that affect the E-Learning system:

* A thorough literature review to collect the dimensions and CSFs of E-Learning.

* Identification of CSFs commonly used in E-Learning research.

* Utilisation of AHP-GDM and FAHP methodologies to assess, evaluate, and prioritise E-Learning CSFs.

Based on the above steps, the selection framework of CSFs is illustrated in Fig 1. The AHP-GDM and FAHP methodologies were employed to prioritize the selected CSFs. Table 1 lists the chosen factors and their dimensions.

[Figure omitted. See PDF.]

Fig 1. AHP-GDM and FAHP based framework for ranking CSFs of E-Learning.

https://doi.org/10.1371/journal.pone.0231465.g001

[Figure omitted. See PDF.]

Table 1. Dimensions and CSFs of E-Learning.

https://doi.org/10.1371/journal.pone.0231465.t001

3.1. Students’ dimension

Students are considered as one of the main stakeholders of the E-Learning system. It has been observed that students are given preference over the other stakeholders as they are the main beneficiary of the E-Learning system [5, 37, 38]. Moreover, students are expected to acquire assistance from the system and the system would be more successful and valuable if students use it appropriately. Meanwhile, students’ demand for diversified education has been growing. For instance, request for such education from non-traditional female students has increased considerably. E-Learning can play a crucial role in fulfilling such demands. Not only, but full-time working professionals may also take advantage of the E-Learning system as per their convenience. Thus, students’ dimension greatly influences the E-Learning system. The various CSFs from this dimension that influences the E-Learning system are listed in Table 1.

3.2. Instructors’ dimension

The instructors’ dimension concerns the satisfaction and gratification of students regarding their E-Learning experience. Instructors’ attitude and approach are critical in the efficient and productive implementation of the system. Moreover, their teaching style also plays a key role and provides a stimulating effect on the student’s recognition of E-Learning education [38]. Students are sure to engage in E-Learning if they find the instructors’ lectures useful and friendly with quality content. Additionally, instructors’ characteristics significantly decide the impact, effectiveness, and quality of educational LMS [39, 40]. Another study has highlighted that the result and conclusion of learning management systems are exaggerated by the teacher’s features, like opinion concerning the use of novel technology, teaching-learning styles, and mechanism of technology employed [41]. Several CSFs under the instructors’ dimensions are presented in Table 1.

3.3. Design and contents’ dimension

The dimension of content design has a direct effect on the success of E-Learning. Well-structured course contents, tools for learning, and activities and curriculum enable positive learning experiences. If a virtual course’s interface is user-friendly and the contents are clear, the learners’ interest and gratification will increase. Simple and properly arranged interfaces of virtual courses encourage students to take up those courses and learn via the Internet, according to their time, place, and flexibility. Many researchers [42, 43] have listed various important content design CSFs, which are mentioned in Table 1.

3.4. System and technological dimension

This dimension assumes a key part in conveying instructive teaching and learning through web technology [44] using different devices like video conferencing, sound, and content-based systems. System quality relates to the E-Learning website’s quality through which students can easily access learning materials of different courses [44]. The innovative system design with changing technological dimension helps in the rapid utilization of the devices [45]. For example, the broadcast speed of web-information impacts the students’ fulfillment, whereby the stacking pace of web-information remains specifically associated with the host server. The stacking pace of the online edge will be high if a preeminent class server is used [46]. Table 1 tabulates the various CSFs under the system and technological dimension.

3.5. Institutional management service dimension

The institutional management service dimension concerns the organizational support feature that makes E-Learning effective. Institutional support is critical in measuring the perceived satisfaction of all stakeholders [38, 47]. Infrastructure readiness, financial readiness, training of users, support for faculty, and ethical and legal issues are important CSFs of this dimension. The CSFs related to the institutional management service dimension are presented in Table 1.

4. Overview of MCDM based research methodologies

The present paper applies two research methodologies i.e. AHP-GDM and FAHP. The AHP has been widely used in solving problems containing multi-level hierarchical structures of objectives, criteria, sub-criteria, and alternatives. Since the decision making in AHP is carried out through the human judgment of DM, there lies a chance of bias decision making. The bias decision making may be restricted by using GDM hence a combinatorial approach of AHP-GDM will be fruitful. Further, the decision making may have vagueness, this may be removed by using fuzzy AHP. AHP-GDM and FAHP provide ease in pairwise comparison and fine-tuning the results, the detail steps are further described as follows:

4.1. Analytic hierarchy process (AHP)

In 1980, T.L. Saaty developed a systematic decision-support procedure known as AHP. The developed AHP process is capable of resolving the complex problems which may consist of multiple-criteria, multiple-levels, complex structure, etc. using a pairwise judgment from DMs. The use of AHP has been found in various researches in a wide variety of research problems [83–85].

The AHP uses the judgment of a DM in a pairwise comparison using Saaty’s nine-point scale as given in Table 2. The DM’s vast experience and deep knowledge related to the decision problem helps in a pairwise comparison. The single opinion from DM may be biased and misleading. Such a decision may not fulfill the requirement of the decision problem and thus becomes unusable in decision-making. The AHP may further be strengthened by using group decision-making (GDM). The use of GDM will improve the accuracy of decision-making. The GDM uses more DMs for resolving the given problem, thus it helps in delivering meaningful, robust and comprehensive single decision to solve the given decision problem.

[Figure omitted. See PDF.]

Table 2. Saaty’s nine-point scale [86].

https://doi.org/10.1371/journal.pone.0231465.t002

The detailed AHP-GDM is discussed as follows:

Step 1:

The various CSFs of E-Learning forming a single hierarchy are converted to a matrix form to construct a comparison matrix ‘Q’. The comparison matrix may also be termed as a decision matrix. In a pairwise comparison, the scale given in Table 2 is be applied to compare a pair of two CSFs presents in the ‘Q’ matrix. Thus, each entry in Q matrix is compared with respective entry to its level of importance. In other words, the element, dmn of the ‘Q’ matrix, compares the mth element with that of the nth in terms of its importance level.

(1)

Step 2:

The ‘Q’ decision matrix is be arrived using a pairwise judgment from each DM participating in the group decision-making process. The overall geometric means (GM) of each pairwise decision is calculated. Thus the pairwise decision and subsequent priority vector (PV) are be calculated.

Step 3:

In the process of AHP-GDM, the ‘Q’ decision matrix with the derived pairwise comparison entries, an overall summation of the product of the sum of each vector column for both the matrices with the PV values of each row is calculated. Later on, the principal eigenvalue (λmax), is be calculated as:(2)where cj is the sum of each column vector.

Step 4:

In the AHP-GDM, the acceptance of decision is based on the decision consistency of DMs. Hence it becomes mandatory to verify the decision consistency in the decision problem. From the derived ‘Q’ decision matrix, the consistency index (CI) is calculated by using Eq (3):(3)Where CI = consistency index and n = the number of elements of each of the matrix.

Step 5:

The consistency ratio is calculated using the random index. The random index (RI) can be calculated using Eq (4).

(4)

Step 6:

The value of consistency ratio (CR) is significant to decide whether the derived a pairwise judgmental matrix is acceptable or not. Based on the CR value the matrix is accepted if the obtained CR value is less than 10%. In the case of the obtained value is more than 10%, the pairwise judgmental decision is revised. The CR may be obtained using Eq (4).

(5)

Step 7:

In a pairwise comparison of decision matrix “Q”, the fuzzy scale based on TFN is used, Which is shown in Table 10. The fuzzy scale may help in getting the pairwise comparison matrices (Ai,i = 1,2,…,n). Based on the importance level, DMs may fix the relevant relationship and suitable choose the scale to assign a weight to these matrices.

4.2. Fuzzy Analytic Hierarchy Process (FAHP)

In the FAHP, a fuzzy set theory along with extension principle can be applied. The use of FAHP is considered indispensable when the inaccuracy in decision-making has to be removed. Many a time in decision-making, manual judgment is critical. However, while making a human judgment there is a possibility of making an error in judgmental decisions. The DM may become bias in his decision making. To reduce such error often the fuzzy methodology is applied in decision-making [87]. The basic fuzzy set theory and extension principles help in accurate decision making. The basic fuzzy set theory and extension principles are further described.

4.2.1. Fuzzy set theory.

The fuzzy set theory helps considerably in robust decision-making. When DM decides a decision in the crisp environment there lies many disadvantages. It does not offer more alternatives to a DM hence sometimes possess judgmental biases and vagueness. The vague or sometimes-incomplete information in crisp form may be misleading. The use of fuzzy set theory can be used in decision-making in a fuzzy environment.

The triangular fuzzy numbers (TFN) (m1, n1, o1) or trapezoidal numbers (m1, n1, o1, p1) can be used in pairwise decision-making [88].

The fuzzy set theory makes use of TFNs for different arithmetic operations [89]. TFNs are represented by P1 and P2 as (c1,d1,e1) and (c2,d2,e2) respectively as shown in Fig 2. TFN can be employed to represent the fuzzy terms in gathering information, further, it can be employed in representing the uncertainty in information, vagueness. In any arithmetic operation, two TFNs are used. The arithmetic operations like subtraction, addition, division, and multiplication may be carried out using two TFNs. Such arithmetic operations can be represented by the following Eqs (4–10):

[Figure omitted. See PDF.]

Fig 2. Triangular fuzzy number (P).

https://doi.org/10.1371/journal.pone.0231465.g002

(6)(7)(8)(9)(10)

4.2.2. Application of extent analysis principle in MCDM under fuzzy environment.

The comparison of two TFNs, is carried out using the extent analysis principles [90]. Two sets i.e. set of objective and set for goal are considered as Y = {y1, y2,………,yn} and Z = {z1, z2,………,z3} respectively. Thus, using the extension principle each object is derived and extent analysis for each goal can be performed. As a result, m extent analysis values for each object can be derived as:(11)

Where are TFNs and represented as (c,d,e). The procedure based on extent analysis [91, 92] is explained below:

Step 1: Establishing a hierarchy structure for the given goal

The E-Learning system can be grouped into multiple levels comprising of dimensions and CSFs. The hierarchy is verified suing feedback from the DMs. It is important to frame the hierarchical structure for ranking. The final hierarchy may have a ranking of dimensions and CSFs of E-Learning at the top, followed by dimensions and CSFs of E-Learning.

Step 2: Establishing the pairwise comparison for dimension and CSFs of E-Learning using TFNs

The dimension and CSFs of E-Learning are analyzed and compared using the feedback from DMs. The pairwise comparison of dimension and CSFs of E-Learning is finally established. In the entire pairwise comparison matrix, the TFNs are used in fixing the relationship among such pairwise comparison.

Step 3: Obtaining the value of fuzzy synthetic extent

(12)

Using fuzzy summation of TFNs, m extent analysis values , are obtained as:(13)and , gives the fuzzy summation of

values are calculated as(14)

The inverse of the vector is obtained as:(15)

Step 4: Obtaining the degree of possibility of supremacy for two TFNs i.e. P2 = (c2,d2,e2)≥P1 = (c1,d1,e1)(16)and can be represented as:(17)(18)Various DMs are involved in the group decision-making for instance K DMs may be participating, thus the subsequent pairwise comparisons yield n elements. A set of K matrices, , where represents the relative importance of element i to j, as derived by DM k. The aggregation is obtained using the Eq (19).

(19)

The two TFNs i.e. (c1, d1, e1) and (c2, d2, e2) intersect at d which as shown in Fig 3. It also gives ordinate d, from the possible highest intersection between two fuzzy numbers μP1 and μP2 denoted as Q. Thus P1 and P2, are calculated through the values of V (P1 ≥ P2) and V (P2 ≥ P1).

[Figure omitted. See PDF.]

Fig 3. The intersection of TFNs [41].

https://doi.org/10.1371/journal.pone.0231465.g003

Step 5: Obtain the degree of possibility for a given convex fuzzy number such that it is greater than k convex

Fuzzy number P1(i = 1,2,….,k) is derived as(20)

Considering,(21)

Weight vector is derived as G′ = (d′(B1),d′(B2),………,d′(Bn))T

Such that Bi(i = 1,2,…..,n) has n elements

Step 6: Obtain the normalized weight vectors.

The normalized weight vector is calculated using Eq (22)(22)

Where C denotes crisp number.

Step 7: Obtaining the overall score of each CSFs dimension and its factors for the prioritization

The overall priority weights of each dimension and CSFs of E-Learning is obtained by multiplying local weight and global weight. The global weights of dimensions and CSFs of E-Learning are arranged in descending order. The overall rank so obtained gives the required prioritization.

5. AHP-GDM and FAHP methodologies for prioritizing the dimensions and CSFs of E-Learning

AHP-GDM and FAHP are MCDM methodology for a pairwise comparison in the crisp environment and fuzzy environment respectively. To evaluate and prioritize the dimensions and CSFs of E-Learning, a combinatorial approach of AHP-GDM and FAHP are used. The AHP-GDM methodology offers accuracy benefits in calculating the weights of dimension and CSFs of E-Learning.

The FAHP helps to eliminate vagueness in the process of decision-making. To carry out AHP-GDM and FAHP, five DMs have been identified who have 6 years or more experience of direct involvement in training through E-Learning. Out of these five DMs, Two DMs also have E-Learning organizational experience with hardware and E-Learning software administration. The other three DMs are having core teaching experience in the E-Learning system. The DMs were persuaded to cooperate without reservation for the cause of educational research and they accepted. After explaining the AHP-GDM method, the questionnaire was given to the expert DMs physically and data was collected from them. Fig 1 describes the framework obtained through a detailed review of the literature for evaluating and prioritizing the E-Learning factors. Five DMs i.e., DM1, DM2, DM3, DM4, and DM5 assessed five dimensions that are given in Tables 3, 4, 5, 6 and 7. The following steps were followed to get the final prioritized list of dimensions and factors.

1. Defining the goal of the problem.

2. Forming hierarchical Structure of the dimensions and its factors.

3. Pairwise comparison matrices construction for each DM.

4. Synthesizing of a pairwise comparison of each dimension and its factors.

5. Check the consistency of the pairwise comparisons

6. Aggregation of DMs’ judgment.

7. Ranking of all dimensions and factors based on global weight.

[Figure omitted. See PDF.]

Table 3. A pairwise comparison of E-Learning dimensions by DM1 using AHP-GDM.

https://doi.org/10.1371/journal.pone.0231465.t003

[Figure omitted. See PDF.]

Table 4. A pairwise comparison of E-Learning dimensions by DM2 using AHP-GDM.

https://doi.org/10.1371/journal.pone.0231465.t004

[Figure omitted. See PDF.]

Table 5. A pairwise comparison of E-Learning dimensions by DM3 using AHP-GDM.

https://doi.org/10.1371/journal.pone.0231465.t005

[Figure omitted. See PDF.]

Table 6. A pairwise comparison of E-Learning dimensions by DM4 using AHP-GDM.

https://doi.org/10.1371/journal.pone.0231465.t006

[Figure omitted. See PDF.]

Table 7. A pairwise comparison of E-Learning dimensions by DM5 using AHP-GDM.

https://doi.org/10.1371/journal.pone.0231465.t007

The output of each step is given below:

Step 1: Goal

The initial step of prioritization of dimension and CSFs of E-Learning are considered as the goal of the problem.

Step 2: Hierarchical Structure

The second step is dedicated to forming the hierarchical structure by establishing the dimensions and relevant factors for each dimension. Fig 4 illustrates the typical hierarchical structure for dimensions and CSFs of the E-Learning system which is developed by arranging the obtained dimensions and CSFs.

[Figure omitted. See PDF.]

Fig 4. Hierarchical structure for dimensions and CSFs of E-Learning system.

https://doi.org/10.1371/journal.pone.0231465.g004

The level I indicates the prioritizing the CSFs of E-Learning. The Five dimensions identified such as Students’ Dimension (SD), Instructors’ Dimension (ID), Design and Content’s Dimension (DCD), System & Technological Dimension (STD), Institutional Management Dimension (IMD) are placed at level II, The various CSFs under different category of dimensions like Students’ Dimension (SD) consist Attitude towards E-Learning (ATE), Students’ Motivation (SM), Interaction with other Students (IOS), Commitment towards Online Studies(CTO), General Internet self-efficacy (GIS). The Instructors’ Dimension (ID) has CSFs of Instructors’ Attitude towards (IAT), Instructors’ ICT skills (IIS), Easy Language Communication (ELC), Appropriate Timely Feedback (ATF). The CSF of Design and Content’s Dimension (DCD) are found to be Appropriate Course Design (ACD), Interactive Learning Activity (ILA), Use of Multimedia Instruction (UMI), User–Friendly Organized (UFO), Understandable Content (UC). The System & Technological Dimension (STD) consist of CSFs like Appropriate System (AS), Ease of Access (EoA), Good Internet Speed (GIS), Technical Support for Users (TSU), Reliability (R), Efficient Technology Infrastructure (ETI), Finally, the Institutional Management Dimension (IMD) possess the CSFs of Infrastructure Readiness (IR), Financial Readiness (FR), Training for User (SST), Support for Faculty (SF), Ethical & Legal Issues (ELI).

Step 3: Pairwise Comparison Matrices

After constructing the hierarchical structure, the relative contribution of each CSFs and their dimensions are obtained by making a pairwise comparison of each DM. In AHP-GDM, an expert’s opinion is used for getting final opinions. The feedback from the DMs’ plays an important role in getting the pairwise comparison. Generally in AHP, a single DM is considered sufficient to provide a decision. However, single DM may some-time gives an ambiguous judgment. To remove this error in decision-making, group decision-making (GDM) has been employed. Five DMs were participated to give their expert opinion in framing a pairwise comparison, which helped in getting a decision without bias. The scale used for a pairwise comparison matrix is depicted in Table 2 [86]. The pairwise comparison matrix Tables 3–7 are further synthesized using the geometric mean method to give the final pairwise comparison matrix. The matrix so obtained using GDM has more accuracy compared to a single DM.

Step 4: Synthesizing of Pairwise Comparison

After completing a pairwise comparison of various dimensions of E-Learning, the results are tabulated as an output matrix (Tables 3–7). It refers to the relative contribution of one element over the other. Since the group decision-making plays an important role in more accurate decision-making. Hence, the pairwise comparison matrix using GDM is synthesized using the geometric mean. The geometric mean method is preferred over the arithmetic mean method due to its non-reciprocity of the pairwise matrix. The pairwise comparison matrix derived by each DM is synthesized and presented in Table 8.

[Figure omitted. See PDF.]

Table 8. Synthesized pairwise comparison of E-Learning dimensions by DM1 to DM5 using GM.

https://doi.org/10.1371/journal.pone.0231465.t008

Step 5: Check Consistency

Consistency level of the obtained values after a pairwise comparisons are checked with the help of Consistency level (CI) and Random Index (RI) using Eq 3.

Step 6: Aggregation of Judgement

The next step was to aggregate the multiple values provided by different DMs in the relevant matrix into a single value. Thus, the weighted GM method was applied to the aggregate judgment of all five DMs. After the single value was obtained the same process was repeated to find the synthesize value of each dimension and factors after aggregation.

Step 7: Ranking

In the final step, the factors from all dimensions were ranked based on their global weights, which was determined as a relative contribution to the E-Learning success. The AHP pairwise matrix offers local weights for each element. The product of such local weight of E-Learning dimension and E-Learning CSFs are obtained using the following relations:

Global weights = ∑(Local weight for dimension i x local weight for factor j with respect to dimension i)

The global weights so obtained are arranged in descending order to prioritize the CSFs of E-Learning that contributes the maximum influence in the E-Learning success. Thus it has been seen that AHP-GDM will help to get an overall ranking of factors [91].

Table 9 shows the composite weights of each E-Learning CSFs attained through AHP-GDM.

[Figure omitted. See PDF.]

Table 9. Composite rank and weight of dimension and CSFs of E-Learning using AHP-GDM.

https://doi.org/10.1371/journal.pone.0231465.t009

In the same way, FAHP is employed to establish the weighs of dimension and CSFs to get the ranking. The TFN based scale as shown in Table 10 has been used in getting the weights of factors and their dimensions. The methodology explained in section 3 is followed to found the local weights and global weight. Table 11 demonstrates the weights of dimensions. Table 12 shows the weight of each factor. The prioritization obtained using MCDM is further compared and shown in Table 13.

[Figure omitted. See PDF.]

Table 10. TFN scale for a pairwise comparison using FAHP [87].

https://doi.org/10.1371/journal.pone.0231465.t010

[Figure omitted. See PDF.]

Table 11. A pairwise comparison of E-Learning dimensions using FAHP.

https://doi.org/10.1371/journal.pone.0231465.t011

[Figure omitted. See PDF.]

Table 12. Composite rank and weight of dimension and CSFs of E-Learning using FAHP.

https://doi.org/10.1371/journal.pone.0231465.t012

[Figure omitted. See PDF.]

Table 13. A comparison of composite weights and rank of dimension and CSFs of E-Learning using AHP-GDM and FAHP.

https://doi.org/10.1371/journal.pone.0231465.t013

6. Results and discussion

The present paper help to establish the influence of dimension and CSFs of E-Learning using MCDM. Both AHP and FAHP methods are capable of delivering the fruitful results. In case of AHP, the results may further be improved using GDM is preferred over individual decision-making. The prioritized CSFs may prove to be helpful to the administrators to derive, monitor and control the E-Learning system successfully. Generally, the E-Learning infrastructure involves huge investment for providing the latest infrastructure which includes software and hardware. To make it successful these CSFs are essentially being controlled. The results obtained by both methods i.e. AHP-GDM and FAHP are compared to get accurate ranking. E-Learning stakeholders such as the institute of higher education may find the prioritization of CSFs useful AHP-GDM and FAHP's combinatorial approach can be used to rank and extract weights. It is useful to compare the weights of all factors with AHP-GDM and FAHP to understand the importance of each CSF.

The found results show the priority of twenty-five factors. The global weights attained are organized in a decreasing manner. The global weights of top five factors thus obtained are: 0.1750 > 0.0919 > 0.0906 > 0.0886 > 0.0733. Where ‘>‘ indicates more preference over the criteria. The CSFs of E-Learning contributing significantly to the success of E-Learning are 1. Financial Readiness (FR) 2. Efficient Technology Infrastructure (ETI) 3. Attitude towards E-Learning (ATE) 4. Infrastructure Readiness (IR) and 5. Reliability (R).

However, based on the FAHP the weights and its group classification are changing. The obtained results shows priority of first five CSFs as: 0.08 > 0.077 >0.069 > 0.063 > 0.056. Where ‘>‘ indicates more preference over criteria. The CSFs of E-Learning with reference to its comparative contribution to the success of E-Learning is found as 1. Financial Readiness (FR) 2. Attitude towards E-Learning (ATE) 3. Infrastructure Readiness (IR) 4. Appropriate Course Design (ACD) 5. Appropriate Timely Feedback (ATF).

Fig 5 demonstrates the comparison of CSFs weights of E-Learning obtained with MCDM. Similarly, it is also possible to compare rank E-Learning factors using MCDM. Fig 6 displays the comparison of E-Learning rankings obtained from MCDM.

[Figure omitted. See PDF.]

Fig 5. A comparison of weights of CSFs of E-Learning using AHP-GDM and FAHP.

https://doi.org/10.1371/journal.pone.0231465.g005

[Figure omitted. See PDF.]

Fig 6. A comparison of ranks CSFs of E-Learning using AHP-GDM and FAHP.

https://doi.org/10.1371/journal.pone.0231465.g006

Based on the weights obtained by MCDM two prioritization sequences are obtained. The various weights of five E-Learning dimensions obtained by MCDM are as: 0.3283 > 0.2603 > 0.1900 > 0.1251 > 0.0964 where ‘>‘ represents ‘more importance for IMD > SD > STD > ID > DCD i.e. Institutional Management Dimension (IMD), Students’ Dimension (SD), System and Technological Dimension (STD), Instructors' Dimension (ID), and Design and Contents’ Dimension (DCD), Similarly, The various weights of five E-Learning dimensions obtained by FAHP methodology are as 0.2393 > 0.2392 > 0.1844 > 0.1791 > 0.1579 > where ‘>‘ represents ‘more importance’ for IMD > SD > DCD > ID > STD i.e. for Institutional Management Dimension (IMD), Students’ Dimension (SD), Design and Contents’ Dimension (DCD), Instructors' Dimension (ID), and System and Technological Dimension (STD).

Similarly, the prioritized rank of CSFs by MCDM is obtained. To systemize the prioritizations obtained by both methodologies, it is further grouped into three categories i.e. high influence, moderate influence, and low influence. Based on the DMs’ expert opinion prioritized CSFs of E-Learning are using AHP-GDM are classified into three groups as shown in Table 14. The Financial Readiness (FR), Efficient Technology Infrastructure (ETI), Attitude towards E-Learning (ATE), Infrastructure Readiness (IR) and Commitment towards Online Studies (CTO) classified as group I having high influence, The Appropriate Timely Feedback (ATF), Appropriate System (AS), Students’ Motivation (SM), Instructors’ Attitude towards E-Learning (IAT), Appropriate Course Design (ACD), Interaction with other Students (IOS), Support for Faculty (SF) are classified as Gr.II having moderate influence. Whereas, the User-Friendly Organized (UFO), Easy Language Communication (ELC), Technical Support for Users (TSU), General Internet self-efficacy (GIS), Training for User (SST), Instructors’ ICT Skills (IIS), Use of Multimedia Instruction (UMI), Good Internet Speed (GIS), Interactive Learning Activity (ILA), Ethical & Legal Issues (ELI), Ease of Access (EoA), Understandable Content (UC), Reliability (R) are classified as low influences group. Similarly, CSFs of E-Learning using FAHP are also grouped which is shown in Table 14.

[Figure omitted. See PDF.]

Table 14. Classification of CSFs of E-Learning into varying degree of influence-group.

https://doi.org/10.1371/journal.pone.0231465.t014

7. Limitations and scope for future research

The assessment and prioritization of E-Learning’s dimensions and CSFs influence its success. Nevertheless, depending on the different financial, social, and regional conditions, the effect of CSFs may change when compared with the global situation. CSFs of E-Learning has been established for Saudi Arabia in the current research after evaluation and prioritization. However, the priorities obtained for 25 CSFs are limited to this country. Besides that, this present research has provided a robust AHP-GDM and FAHP based methodology for assessing and prioritizing the CSFs. Nonetheless, the results may not be generalized, as the assessment was based on DMs. By employing GDM, the biases have been removed to a great extent; however, a more accurate judgment is required. Hence, TOPSIS and Fuzzy TOPSIS may be employed to remove vagueness and biases in judgment and optimize the CSFs that influence E-Learning.

8. Conclusions

E-Learning and mobile learning (M-Learning) are the subset of digital learning (D-Learning). These days, students are well conversant with computer and mobile usage hence find both the version easy and comfortable for their learning. Both E-Learning and M-Learning have their users varying with age, sex, region, social culture etc. The E-Learning provides speedy access with multimedia to demonstrate much more understanding to users. The success of E-Learning is highly depending upon the CSFs. The CSFs affecting the E-Learning success are many hence it is essential to evaluate and prioritize them so that the management providing E-Learning can invest, regulate E-Learning infrastructure in an effective manner.

As it is evident that, CSFs play a key role in E-Learning success. Hence, investigations related to the impact of dimensions and CSFs on learning and teaching are mandatory. After assessing the impact of each CSF, the various stakeholders, such as university authority, students, and instructors, will be able to control the negative effects of each of these E-Learning factors and their dimensions. The MCDM approach could be fruitful in categorizing each dimension and CSF of the E-Learning system. This categorization of factors will help stakeholders to decide the strategic policy for the resource management of money and time in creating and enhancing service infrastructure to improve the teaching-learning process.

9. Recommendation of effective E-Learning implementation for students, instructors and developers

It is recommended to evaluate and control CSFs of E-Learning in an effective manner by utilizing resources (Hardware and Software). It is further recommended to provide proper training of the E-Learning system to students and instructors to obtain learning and teaching skills respectively. Lack of training can prevent effective utilization and possible benefits from the system. Professional training can enhance the instructors’ capabilities to access virtual class facilities. It helps in increasing multimedia in the E-Learning. The instructor will be able to manage various miscellaneous technical applications like assignments and tests, discussion board, course messages, and grading for effective course management. The well-trained students can achieve full benefits of the system. Instructors can also adopt suitable teaching practices to provide a rich learning environment by introducing teaching notes, video, and multimedia. Instructors can also use social media to enhance their alliance with students. They can allow them to share their E-Learning system experience through various social media, such as Facebook, Twitter or YouTube.

The developers in consultation with instructors must develop various learning activities to enhance alliance between instructors and students. They can also develop purposeful and effective learning activities that can motivate the stakeholders to use the system effectively. In order to enhance the interest of students, instructors and other users, audio, video and multimedia content can be imbibed in the E-Learning. The developers may develop user-friendly environment by providing an easy and handy applications like emails, course messages, frequently asked questions (FAQs).

Universities can manage to provide special budgets for E-Learning units to purchase new or upgrade technological infrastructure. The installation of enhanced high-speed internet bandwidth of the E-Learning system can provide ease in usage to the full extent. Furthermore, upgraded software, sufficient laboratories with high computer inventory with uninterrupted networking amenities will boost the motivation of students and instructors.

Citation: Naveed QN, Qureshi MRN, Tairan N, Mohammad A, Shaikh A, Alsayed AO, et al. (2020) Evaluating critical success factors in implementing E-learning system using multi-criteria decision-making. PLoS ONE 15(5): e0231465. https://doi.org/10.1371/journal.pone.0231465

References

1. Zare M., et al., “Multi-criteria decision making approach in E-learning: A systematic review and classification,” Appl. Soft Comput., vol. 45, pp. 108–128, 2016.

2. Samsudeen S. N. and Mohamed R., “University students’ intention to use e-learning systems: A study of higher educational institutions in Sri Lanka,” Interact. Technol. Smart Educ., 2019.

3. Alhabeeb A. and Rowley J., “E-learning critical success factors: Comparing perspectives from academic staff and students,” Comput. Educ., vol. 127, no. October 2017, pp. 1–12, 2018.

4. Rasheed R. A., Kamsin A., and Abdullah N. A., “Challenges in the online component of blended learning: A systematic review,” Comput. Educ., vol. 144, p. 103701, 2020.

5. Al-Fraihat D., Joy M., Sinclair J., and others, “Evaluating E-learning systems success: An empirical study,” Comput. Human Behav., vol. 102, pp. 67–86, 2020.

6. Rohayani A. H. H. and others, “A Literature Review: Readiness Factors to Measuring e-Learning Readiness in Higher Education,” Procedia Comput. Sci., vol. 59, pp. 230–234, 2015.

7. Rajab K. D., “The effectiveness and potential of E-learning in war zones: An empirical comparison of face-to-face and online education in Saudi Arabia,” IEEE Access, vol. 6, pp. 6783–6794, 2018.

8. Wang T.-H., “Developing Web-based assessment strategies for facilitating junior high school students to perform self-regulated learning in an e-Learning environment,” Comput. Educ., vol. 57, no. 2, pp. 1801–1812, 2011.

9. Vanitha V., Krishnan P., and Elakkiya R., “Collaborative optimization algorithm for learning path construction in E-learning,” Comput. Electr. Eng., vol. 77, pp. 325–338, 2019.

10. Navimipour N. J. and Zareie B., “A model for assessing the impact of e-learning systems on employees satisfaction,” Comput. Human Behav., vol. 53, pp. 475–485, 2015.

11. Tawafak R. M., Romli A. B., and Arshah R. B. A., “Continued Intention to Use UCOM: Four Factors for Integrating With a Technology Acceptance Model to Moderate the Satisfaction of Learning,” IEEE Access, vol. 6, pp. 66481–66498, 2018.

12. Quadri N. N., Muhammed A., Sanober S., Qureshi M. R. N., and Shah A., “Barriers Effecting Successful Implementation of E-Learning in Saudi Arabian Universities,” Int. J. Emerg. Technol. Learn., vol. 12, no. 06, pp. 94–107, 2017.

13. Al Gamdi M. A. and Samarji A., “Perceived Barriers towards e-Learning by Faculty Members at a Recently Established University in Saudi Arabia,” Int. J. Inf. Educ. Technol., vol. 6, no. 1, p. 23, 2016.

14. Naveed Q. N., Muhammad A., Sanober S., Qureshi M. R. N., and Shah A., “A Mixed Method Study for Investigating Critical Success Factors (CSFs) of E-Learning in Saudi Arabian Universities,” Int. J. Adv. Comput. Sci. Appl., vol. 8, no. 5, pp. 171–178, 2017.

15. Ahmad N., Quadri N., Qureshi M., and Alam M., “Relationship Modeling of Critical Success Factors for Enhancing Sustainability and Performance in E-Learning,” Sustainability, vol. 10, no. 12, p. 4776, 2018.

16. Q. N. Naveed, M. R. N. Qureshi, A. O. Alsayed, N. Ahmad, S. Sanober, and A. Shah, “Assimilating E-Learning barriers using an interpretive structural modeling (ISM),” 2017 4th IEEE Int. Conf. Eng. Technol. Appl. Sci., pp. 1–7, 2017.

17. Bhuasiri W., Xaymoungkhoun O., Zo H., Rho J. J., and Ciganek A. P., “Critical success factors for e-learning in developing countries: A comparative analysis between ICT experts and faculty,” Comput. {&} Educ., vol. 58, no. 2, pp. 843–855, 2012.

18. Rockart J. F., “Chief executives define their own data needs.,” Harv. Bus. Rev., vol. 57, no. 2, pp. 81–93, 1979. pmid:10297607

19. Noorulhasan Naveed N. A., “Critical Success Factors (CSFs) for Cloud-Based e-Learning,” Int. J. Emerg. Technol. Learn., vol. 14, no. 1, pp. 140–149, 2019.

20. Alfoqahaa S., “Critical success factors of small and medium-sized enterprises in Palestine,” J. Res. Mark. Entrep., vol. 20, no. 2, pp. 170–188, 2018.

21. Lagzian F., Abrizah A., and Wee M. C., “Critical success factors for institutional repositories implementation,” Electron. Libr., vol. 33, no. 2, pp. 196–209, 2015.

22. Yassine A. F. A., Battou A., and Omar B. A. Z., “Learning Through Massive Open Online Courses Platforms Based on Fuzzy Analytic Hierarchy Process,” Int. J. Smart Educ. Urban Soc., vol. 10, no. 3, pp. 1–12, 2019.

23. Choi C.-R. and Jeong H.-Y., “Quality evaluation for multimedia contents of e-learning systems using the ANP approach on high speed network,” Multimed. Tools Appl., pp. 1–23, 2019.

24. Kannchen M. and Ziemba P., “Use of the PVM Method Computed in Vector Space of Increments in Decision Aiding Related to Urban Development,” Symmetry (Basel)., vol. 11, no. 4, 2019.

25. Mohammed H. J., Kasim M. M., and Shaharanee I. N., “Evaluation of E-Learning Approaches Using AHP-TOPSIS Technique,” J. Telecommun. Electron. Comput. Eng., vol. 10, no. 1–10, pp. 7–10, 2018.

26. Anggrainingsih R., Umam M. Z., and Setiadi H., “Determining e-learning success factor in higher education based on user perspective using Fuzzy AHP,” in MATEC Web of Conferences, 2018, vol. 154, p. 3011.

27. Garg R., Kumar R., and Garg S., “MADM-Based Parametric Selection and Ranking of E-Learning Websites Using Fuzzy COPRAS,” IEEE Trans. Educ., vol. 62, no. 1, pp. 11–18, 2018.

28. Jain D., Garg R., Bansal A., and Saini K. K., “Selection and ranking of E-learning websites using weighted distance-based approximation,” J. Comput. Educ., vol. 3, no. 2, pp. 193–207, 2016.

29. Garg R., “E-learning website evaluation and selection using multi-attribute decision making matrix methodology,” Comput. Appl. Eng. Educ., vol. 25, no. 6, pp. 938–947, 2017.

30. Garg R., “Optimal selection of E-learning websites using multiattribute decision-making approaches,” J. Multi-Criteria Decis. Anal., vol. 24, no. 3–4, pp. 187–196, 2017.

31. Garg R. and Jain D., “Fuzzy multi-attribute decision making evaluation of e-learning websites using FAHP, COPRAS, VIKOR, WDBA,” Decis. Sci. Lett., vol. 6, no. 4, pp. 351–364, 2017.

32. Turan H., “Assessment factors affecting e-learning using fuzzy analytic hierarchy process and SWARA,” Int. J. Eng. Educ., vol. 34, no. 3, pp. 915–923, 2018.

33. Yigit T., Isik A. H., and Ince M., “Web-based learning object selection software using analytical hierarchy process,” IET Softw., vol. 8, no. 4, pp. 174–183, 2014.

34. W. Li, X. Gao, and G. Fu, “Fuzzy comprehensive assessment of network environment and learning quality combined with the Analytic Hierarchy Process,” in Consumer Electronics, Communications and Networks (CECNet), 2012 2nd International Conference on, 2012, pp. 2600–2603.

35. T.-H. Kang, S. Jia-Horng, S.-Y. Kuo, and S.-M. Wu, “The study for selecting the consignment performance of e-learning of technology college,” in Multimedia Technology (ICMT), 2011 International Conference on, 2011, pp. 3285–3288.

36. Sayed M. and Baker F., “Blended Learning Barriers: An Investigation, Exposition and Solutions,” J. Educ. Pract., vol. 5, no. 6, pp. 81–85, 2014.

37. XaymoungNhoun O., Bhuasiri W., Rho J. J., Zo H., and Kim M.-G., “The critical success factors of e-learning in developing countries,” Korea, vol. 305, p. 701, 2012.

38. Previtali P. and Scarozza D., “Blended learning adoption: a case study of one of the oldest universities in Europe,” Int. J. Educ. Manag., vol. 33, no. 5, pp. 990–998, 2019.

39. Selim H. M., “Critical success factors for e-learning acceptance: Confirmatory factor models,” Comput. Educ., vol. 49, no. 2, pp. 396–413, 2007.

40. Ozkan S. and Koseler R., “Multi-dimensional students’ evaluation of e-learning systems in the higher education context: An empirical investigation,” Comput. Educ., vol. 53, no. 4, pp. 1285–1296, 2009.

41. Sun P.-C., Tsai R. J., Finger G., Chen Y.-Y., and Yeh D., “What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction,” Comput. Educ., vol. 50, no. 4, pp. 1183–1202, 2008.

42. Arora R. and Chhabra I., “Extracting components and factors for quality evaluation of e-learning applications,” in Engineering and Computational Sciences (RAECS), 2014 Recent Advances in, 2014, pp. 1–5.

43. Raspopovic M., Jankulovic A., Runic J., and Lucic V., “Success factors for e-learning in a developing country: A case study of Serbia,” Int. Rev. Res. Open Distrib. Learn., vol. 15, no. 3, 2014.

44. Chopra G., Madan P., Jaisingh P., and Bhaskar P., “Effectiveness of e-learning portal from students’ perspective: A structural equation model (SEM) approach,” Interact. Technol. Smart Educ., vol. 16, no. 2, pp. 94–116, 2019.

45. Malik M. W., “Factor Effecting Learner’s Satisfaction Towards E-Learning: A Conceptual Framework,” OIDA Int. J. Sustain. Dev., vol. 2, no. 3, pp. 77–82, 2010.

46. Y. F. A. Wibowo and K. A. Laksitowening, “Redefining e-learning readiness model,” in Information and Communication Technology (ICoICT), 2015 3rd International Conference on, 2015, pp. 552–557.

47. Ali M., Raza S. A., Qazi W., and Puah C.-H., “Assessing e-learning system in higher education institutes: Evidence from structural equation modelling,” Interact. Technol. Smart Educ., vol. 15, no. 1, pp. 59–78, 2018.

48. Menchaca M. P. and Bekele T. A., “Learner and instructor identified success factors in distance education,” Distance Educ., vol. 29, no. 3, pp. 231–252, 2008.

49. Bhuasiri W., Xaymoungkhoun O., Zo H., Rho J. J., and Ciganek A. P., “Critical success factors for e-learning in developing countries: A comparative analysis between ICT experts and faculty,” Comput. Educ., vol. 58, no. 2, pp. 843–855, 2012.

50. FitzPatrick T., “Key Success Factors of eLearning in Education: A Professional Development Model to Evaluate and Support eLearning.,” US-China Educ. Rev., vol. 9, pp. 789–795, 2012.

51. Frimpon M. F., “A re-structuring of the critical success factors for e-learning deployment,” Am. Int. J. Contemp. Res., vol. 2, no. 3, pp. 115–127, 2012.

52. Kanwal F. and Rehman M., “Factors affecting e-learning adoption in developing countries-empirical evidence from Pakistans higher education sector,” IEEE Access, vol. 5, pp. 10968–10978, 2017.

53. Andersson A., “Seven major challenges for e-learning in developing countries: Case study eBIT, Sri Lanka,” Int. J. Educ. Dev. using ICT, vol. 4, no. 3, 2008.

54. M. Mosakhani and M. Jamporazmey, “Introduce critical success factors (CSFs) of elearning for evaluating e-learning implementation success,” in Educational and Information Technology (ICEIT), 2010 International Conference on, 2010, vol. 1, pp. V1—224.

55. M. R. Mehregan, M. Jamporazmey, M. Hosseinzadeh, and M. Mehrafrouz, “Proposing an approach for evaluating e-learning by integrating critical success factor and fuzzy AHP,” in International Conference on Innovation, Management and Service, Singapore, 2011.

56. Lin C.-C., Ma Z., and Lin R. C.-P., “Re-examining the Critical Success Factors of e-learning from the EU perspective,” Int. J. Manag. Educ., vol. 5, no. 1, pp. 44–62, 2011.

57. Ordóñez A. C., “Predicting International Critical Success Factors in e-learning,” Universitat Oberta de Catalunya, 2014.

58. Musa M. A. and Othman M. S., “Critical success factor in e-Learning: an examination of technology and student factors,” Int. J. Adv. Eng. Technol., vol. 3, no. 2, p. 140, 2012.

59. N. Laily, A. Kurniawati, and I. A. Puspita, “Critical success factor for e-learning implementation in Institut Teknologi Telkom Bandung using Structural Equation Modeling,” in Information and Communication Technology (ICoICT), 2013 International Conference of, 2013, pp. 427–432.

60. Bahramnezhad F., Asgari P., Ghiyasvandian S., Shiri M., and Bahramnezhad F., “The Learners’ Satisfaction of E-learning: A Review Article,” Am. J. Educ. Res., vol. 4, no. 4, pp. 347–352, 2016.

61. Salloum S. A., Alhamad A. Q. M., Al-Emran M., Monem A. A., and Shaalan K., “Exploring Students Acceptance of E-Learning Through the Development of a Comprehensive Technology Acceptance Model,” IEEE Access, vol. 7, pp. 128445–128462, 2019.

62. Goi C. L. and Ng P. Y., “E-learning in Malaysia: Success factors in implementing e-learning program,” Int. J. Teach. Learn. High. Educ., vol. 20, no. 2, pp. 237–246, 2009.

63. W. Premchaiswadi, P. Porouhan, and N. Premchaiswadi, “An empirical study of the key success factors to adopt e-learning in Thailand,” in Information Society (i-Society), 2012 International Conference on, 2012, pp. 333–338.

64. Cidral W. A., Oliveira T., Di Felice M., and Aparicio M., “E-learning success determinants: Brazilian empirical study,” Comput. Educ., vol. 122, no. February 2017, pp. 273–290, 2018.

65. Puri G., “Critical success Factors in e-Learning—An empirical study,” Int. J. Multidiscip. Res., vol. 2, no. 1, pp. 149–161, 2012.

66. Arora R., et al., “Critical factors for effective eLearning,” Comput. Educ., vol. 3, no. 2, p. 147, 2010.

67. Liaw S.-S., Huang H.-M., and Chen G.-D., “Surveying instructor and learner attitudes toward e-learning,” Comput. Educ., vol. 49, no. 4, pp. 1066–1080, 2007.

68. S. Yong-gui, L. Pu, H. Chunping, and C. Nai, “What Drives a Successful E-Learning? A Comparative Research between China Mainland and Taiwan,” in Intelligent Information Technology Application Workshops, 2008. IITAW’08. International Symposium on, 2008, pp. 961–966.

69. Alamin H. A. A. and Elgabar E. E. A., “Success Factors for Adopting E-learning Application in Sudan,” Int. J. Soft Comput. Eng., vol. 3, no. 6, 2014.

70. Fresen J. W., “Factors influencing lecturer uptake of e-learning,” Teach. English with Technol., vol. 11, no. 1, pp. 81–97, 2011.

71. Liaw S.-S., “Investigating students perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the Blackboard system,” Comput. Educ., vol. 51, no. 2, pp. 864–873, 2008.

72. N. Parsazadeh, N. M. M. Zainuddin, R. Ali, and A. Hematian, “A REVIEW ON THE SUCCESS FACTORS OF E-LEARNING,” in The Second International Conference on e-Technologies and Networks for Development, 2013, pp. 42–49.

73. Lee-Post A., “e-Learning Success Model: An Information Systems Perspective.,” Electron. J. e-learning, vol. 7, no. 1, pp. 61–70, 2009.

74. AbuSneineh W. and Zairi M., “An evaluation framework for E-learning effectiveness in the Arab World,” Int. Encycl. Educ., pp. 521–535, 2010.

75. K. Sigama, B. M. Kalema, and R. M. Kekwaletswe, “Utilizing Web 2.0 and Free Open Source Software to advance e-learning in developing countries,” in Sustainable e-Government and e-Business Innovations (E-LEADERSHIP), 2012 e-Leadership Conference on, 2012, pp. 1–7.

76. Cheawjindakarn B., Suwannatthachote P., Theeraroungchaisri A., and others, “Critical success factors for online distance learning in higher education: A review of the literature,” Creat. Educ., vol. 3, no. 08, p. 61, 2013.

77. Keramati A., Afshari-Mofrad M., and Kamrani A., “The role of readiness factors in E-learning outcomes: An empirical study,” Comput. Educ., vol. 57, no. 3, pp. 1919–1929, 2011.

78. Yew O. F. and Jambulingam M., “Critical Success Factors of E-learning Implementation at Educational Institutions,” J. Interdiscip. Res. Educ. Vol., vol. 5, no. 1, 2015.

79. E. Ansong, S. L. Boateng, R. Boateng, and J. Effah, “Determinants of E-Learning Adoption in Universities: Evidence from a Developing Country,” in 2016 49th Hawaii International Conference on System Sciences (HICSS), 2016, pp. 21–30.

80. Stacey E. and Gerbic P., “Success factors for blended learning,” Hello! Where are you Landsc. Educ. Technol. Proc. ascilite Melb. 2008, pp. 964–968, 2008.

81. Sridharan B., Deng H., and Corbitt B., “Critical success factors in e-learning ecosystems: a qualitative study,” J. Syst. Inf. Technol., vol. 12, no. 4, pp. 263–288, 2010.

82. OdunaiNe S. A., Olugbara O. O., and Ojo S. O., “E-learning Implementation Critical Success Factors,” innovation, vol. 3, p. 4, 2013.

83. Muhammad A., Shaikh A., Naveed Q. N., and Qureshi M. R. N., “Factors Affecting Academic Integrity in E-Learning of Saudi Arabian Universities. An Investigation Using Delphi and AHP,” IEEE Access, vol. 8, pp. 16259–16268, 2020.

84. Garg R., Rahman Z., Qureshi M. N., and Kumar I., “Identifying and ranking critical success factors of customer experience in banks: An analytic hierarchy process (AHP) approach,” J. Model. Manag., vol. 7, no. 2, pp. 201–220, 2012.

85. Talib F., Rahman Z., and Qureshi M. N., “Prioritising the practices of total quality management: An analytic hierarchy process analysis for the service industries,” Total Qual. Manag. Bus. Excell., vol. 22, no. 12, pp. 1331–1351, 2011.

86. Saaty T. L., “Decision making with the analytic hierarchy process,” Int. J. Serv. Sci., vol. 1, no. 1, pp. 83–98, 2008.

87. Q. N. Naveed, M. R. N. Qureshi, A. O. Alsayed, A. Muhammad, S. Sanober, and A. Shah, “Prioritizing Barriers of E-Learning for Effective Teaching-Learning using Fuzzy Analytic Hierarchy Process (FAHP),” in 4th IEEE International Conference on Engineering Technologies and Applied Sciences (ICETAS) 2017, 2017.

88. Kahraman C., Cebeci U., and Ruan D., “Multi-attribute comparison of catering service companies using fuzzy AHP: The case of Turkey,” Int. J. Prod. Econ., vol. 87, no. 2, pp. 171–184, 2004.

89. Kauffman A. and Gupta M. M., “Introduction to Fuzzy Arithmetic, Theory and Application.” Van Nostrand Reinhold, New York, 1991.

90. Chang D.-Y., “Applications of the extent analysis method on fuzzy AHP,” Eur. J. Oper. Res., vol. 95, no. 3, pp. 649–655, 1996.

91. Naveed Q. N., Qureshi M. N., Shaikh A., Alsayed A. O., Sanober S., and Mohiuddin K., “Evaluating and Ranking Cloud-based E-Learning Critical Success Factors (CSFs) using Combinatorial Approach,” IEEE Access, p. 1, 2019.

92. Qureshi M. N., Kumar P., and Kumar D., “Framework for benchmarking logistics performance using fuzzy AHP,” Int. J. Bus. Perform. Supply Chain Model., vol. 1, no. 1, pp. 82–98, 2009.