Benchmarking in Higher Education Using Data Envelopment Analysis and the Bologna Process Data

Abstract

Quality Assurance is one of the core elements of the Bologna Process in European Higher Education and the Bologna Process data of the programs can serve as a rich resource for assessing program quality and benchmarking since the data follow a standardized framework for any program. Despite the standardized framework, the data of each program require an organization in order to come up with comprehensive measures so that such measures can become comparable between programs. This paper aims to propose an analytical approach for academic program quality assessment utilizing the Bologna declarations of the programs. First, we propose a methodology on data organization. Second, the organized data are used for assessments at program and subdivision levels via Data Envelopment Analysis. We illustrate our proposed methodology in an application to Business bachelor degree programs in Turkish universities. We discuss the implications to various stakeholders of higher education from policy makers to students.

Headnote

Abstract. Quality Assurance is one of the core elements of the Bologna Process in European Higher Education and the Bologna Process data of the programs can serve as a rich resource for assessing program quality and benchmarking since the data follow a standardized framework for any program. Despite the standardized framework, the data of each program require an organization in order to come up with comprehensive measures so that such measures can become comparable between programs. This paper aims to propose an analytical approach for academic program quality assessment utilizing the Bologna declarations of the programs. First, we propose a methodology on data organization. Second, the organized data are used for assessments at program and subdivision levels via Data Envelopment Analysis. We illustrate our proposed methodology in an application to Business bachelor degree programs in Turkish universities. We discuss the implications to various stakeholders of higher education from policy makers to students.

Keywords: benchmarking; Data Envelopment Analysis; higher education; Bologna process; quality assurance

(ProQuest: ... denotes formulae omitted.)

1.Introduction

The Bologna Process in European Higher Education aims at the establishment of a common, more competitive and a transparent higher educational area for the member states. Launching of the program in 1999 brought out a standardized framework for measurement of educational contribution. Program quality assurance is one of the core elements of the Bologna Process. Therefore, measurement of the program quality throughout the European educational area is a remarkable area of interest. Any measurement would require valid data and the Bologna Process framework provides valuable information for that. In the scope of the process, every higher education program in every member state has identically organized a chain of data. The path to create an overall impact begins with the outcomes of the courses at the bottom, it fosters the National Qualifications Framework and at the top level sustains the Framework for Qualifications of the European Higher Education Area. This means that a vast amount of information on the quantity and the quality of the higher education programs in Europe is available in a consistent manner. Such quantitative data can be assisted in evaluation of educational programs from different, perspectives targeting at various stakeholders from educational policy makers to students.

The motivation of this paper arises from the idea that an analysis framework for assessing quality of the academic programs can be established and for this purpose, the Bologna Process data of higher education programs can serve as perfect indicators since there exists a homogeneous structure for any program. However, for those to analyze Bologna data of the higher education programs, there exists a challenge to face with. Although the program qualification declarations of each program (prepared by the academic departments pursuing them) follow a standardized framework, the data set of each program is still dispersed and requires an organization in order to come up with comprehensive aggregated measures at the program level so that such measures can become comparable between programs. The paper aims to provide an analytical approach for academic program quality assessment and benchmarking through utilizing the Bologna Process data of the programs. The intended method is the Data Envelopment Analysis (DEA), which is a well-established relative performance measurement methodology to assess units pursuing similar activities [8]. A critical step to fulfil our objective is to come up with aggregate measures from the Bologna data so that program outcomes are comparable. Thus, an important question arises: How can we aggregate and standardize the Bologna data of the programs? The paper primarily seeks for an answer to this question. The contributions of the paper arc twofold. First, it proposes a novel methodology on data organization for Bologna data of the academic programs. Second, the Bologna data is first time used with DEA for academic program quality assessment and benchmarking purposes at both program and subdivision levels. We illustrate our proposed methodology in an application to Business bachelor degree programs with the Bologna Process accreditation carried by Turkish universities. Although the sample includes Business programs, the proposed methodology can be adapted to any other program and it enables cross comparisons between academic programs.

The organization of the paper is as follows: Section 2 provides the basic concepts of the Bologna Process with key definitions. Section 3 introduces the proposed data organization methodology for the program qualification data. Section 4 is devoted to DEA methodology and model design. Section 5 presents the results for the analysis. Section 6 provides a critical review of the data collection process in Bologna and the potential ways to improve proposed methodology. Section 7 concludes with the policy implications for potential stakeholders.

2.The Bologna Process and Framework of Qualifications

The foundation of the Bologna Process stands upon the Sorbonne Declaration signed in 1998 by ministers of France, Germany, UK and Italy. It is followed by Bologna Declaration in 1999 signed by 30 countries resulting in the establishment of European Higher Education Area (EHEA), aiming a common, more competitive and a transparent higher educational area for the member states to provide coherence throughout the higher education programs and to resolve common problems with common policies. Following the Bologna Declaration, Ministerial Conferences have been held in every two years (European Higher Education Area History [10]). Turkey became a member in 2001. The adaptation and continuity of the program is enforced by Higher Education Council of Turkey. One of the key concepts of the Bologna Process is the National Qualifications Framework (NQF). NQF is described as: "The single description, at national level or level of an education system, which is internationally understood and through which all qualifications and other learning achievements in higher education mag be described and related to each other in a coherent way and which defines the relationship between higher education qualifications" (sec European Higher Education Area Glossary [9]). It provides standardized descriptions for the academic programs so that the qualifications gained in any program throughout the member higher education institutions can be comparable.

The framework defined in the Bologna Process is a bottom up process that begins from the individual courses at the bottom resulting a general framework for the EHEA at the top. As a part of the Bologna Process, every academic program adapts its curriculum to the European Credit Transfer and Accumulation System (ECTS). ECTS brings comparability and mobility between academic programs. ECTS credit for a course in an academic program is calculated relying on the workload projected both in and out of the class to succeed the given course. Also, for each course, a matrix that matches the course learning outcomes with the program learning outcomes is identified. Therefore, each course in each academic program has an ECTS credit together with the contribution of the course to the learning outcomes of the program as a whole. Program outcomes are associated with the NQF depending on the field (social sciences, business and administrative sciences, engineering, medicine etc.). NQF for member states builds up to Framework for Qualifications of the European Higher Education Area at the top level. Quality assessment in the scope of the Bologna Process mostly relies on The Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESC) [29] defining a set of common framework for quality at different levels (first published in 2005).

The adaptation and implementation of the Bologna Process together with evolution of quality assurance standards have been heavily studied in the higher education literature. The research can be thought in two groups: overall European higher education evaluations ([2], [12], [14], [15], [18], [21], [22], [27], [30]) and country-based evaluations ([3] in Germany, [20] in Finland, [25] in Hungary, [26] in the Eastern Europe; [28] in Greece, [31] in Spain). The argumentations provided in the papers are at both macro and micro levels. Most of the papers involve discussion of the standards and recommendations for improvement. They usually question the robustness of the system and assess the effects of the Bologna Process on higher education in Europe. The research including empirical analysis, especially in countrybasis, predominantly rely on the data collected via surveys or structured interviews. At this point, current paper proposes a differing tool in order to provide a new insight in quality assessment.

3.Proposed Data Organization Methodology

The Bologna Process data for academic programs follow a specific framework, however, an organization is required for comparability at the program level. In this section, we aim to answer the question: How can we organize the Bologna data of the programs? For a better understanding, we illustrate the data organization methodology over the Business Administration undergraduate degree program provided by Hacettepe University, Department of Business Administration in Ankara, Turkey. The proposed methodology is applicable to any program data and generalizations are provided following the illustration.

3.1.Illustration for Data Organization

Hacettepe University Business Administration Program (HUBAP) is a Bologna Process accredited program with the relevant data published publicly. The department (as any other business department in Turkey) offers courses from 5 main subdivisions. Each subdivision is responsible for a number of compulsory and optional (elective) courses. The numbers of compulsory and optional courses offered by subdivisions in HUBAP arc presented in Tabic 1. An academie program to be accredited to the Bologna Process, one of the several requirements is to list the learning outcomes. Learning outcomes identify different knowledge, skills and competencies, a student graduating from that academic program will gain. HUBAP lias 11 learning outcomes (Hacettepe University Bologna Information Package [13]). The learning outcomes can differ in number and in content from one program to another. For instance, Economics program package of the same university declares 12 outcomes. For each course in the program, the teaching staff declares a 1 to 5 contribution value revealing to wliat degree that specific course contributes to the learning outcomes of the program. Table 2 presents the relationship between 3 example courses in HUBAP and learning outcomes of the program.

Each member state in the Bologna Process has National Qualifications Framework (NQF) determined by the national authority of the given state. Each program under each national authority should match its learning outcomes with this framework depending on its main field (social sciences, business and administrative sciences, engineering, medicine etc.). NQF for Higher Education in Turkey (NQF-HETR) present 6 main qualifications for Business and Administrative Sciences field degree programs. Eaeli of these qualifications lias sub-descriptions (see NQF-HETR [24]). Talile 3 presents the relationship between HUBAP learning outcomes and NQF-HETR qualifications in business and administrative sciences field. The value "1" indicates that the learning outcome matches with the given NQF-HETR qualification.

The relationship tallies between learning outcomes of a program and NQF outcomes is key in the data standardization process. As mentioned above, different programs (even from the same subject area) may be declared as producing different, number of learning outcomes. However, they arc all relate to NQF framework at the end. This enables us to aggregate the data of all courses in all programs to one common ground defined at the national level. Given the information in Tallies 2 and 3, each course in the HUBAP can lie associated with NQF-HETR qualifications. The same is valid for any business program in Turkey even if it has different, number of learning outcomes since the program also lias such relationship tallies. Nevertheless, the same could lie done for Economies program (which has 12 learning outcomes) courses as well and the measures would lie comparable since both Business Administration and Economics programs are subject to same NQF defined for Business and Administrative Sciences field. Last but not least, it is possible to produce aggregate measures for programs in any main field with the same approach, for example: engineering, because all engineering programs are subject, to NQF for engineering field.

NQF-HETR qualifications can ire thought in three main categories: knowledge, skills and competence. Using the relationship data of each course with program outcomes, we calculate knowledge, skills and competence scores for each course revealing their contribution to these three main categories. To illustrate, for Financial Management I course, weighted average scores arc presented in Table 4. For caeli course, 3 scores devoted to knowledge, skills and competence are calculated accordingly. Among HUBAP learning outcomes, articles 6, 9 and 10 are associated with knowledge (see Table 3). Financial Management I contributes to HUBAP learning outcomes 6, 9 and 10 with scores of 3, 5 and 3 (out of 5), respectively (sec Tabic 2). The average score of Financial Management I for knowledge qualification is 3.67. For the NQFHETR qualifications with sub-articles, the method is applied with a similar maimer. Learning outcome 5 of HUBAP is associated with abilities sub-article 1 and learning outcomes of HUBAP 2, 3, 6 and 8 are associated with skills sub-article 2. Financial Management I contributes to outcomes 2, 3, 5, 6 and 8 with scores of 4, 5, 2, 3 and 5, respectively. The average score for this course dedicated to skills is 3.80.

Having the knowledge, skills and competence scores for all courses held in HUBAP and knowing that courses are offered by different subdivisions in the department, subdivision contributions to 3 categories are calculated by averaging the scores of courses associated with each subdivision. Table 5 presents the contribution scores of compulsory and optional courses together with overall subdivision scores for Accounting fe Finance subdivision.

As given in Table 1, each subdivision is responsible for different numbers of compulsory and optional courses. Such a distinction can also be considered while calculating the contributions to NQFHETR categories. Table 6 summarizes the scores for each subdivision relying on the compulsory and optional courses of the given subdivision. Overall contributions arc also provided. Note that the overall contribution scores are not exact averages of scores of each subdivision. This is because some service courses such as offered by outside departments are also accounted in the calculation of overall contributions.

3.2.Generalizations for Data Organization

The proposed methodology is applicable to any program accredited to the Bologna Process regardless of the number of learning outcomes, number of subdivisions or the field which the program belongs to. It results in comparable measures for quality assessment due to the wcll-cstablislicd nature of the Bologna Process declarations of the programs. Suppose there exists a degree program with о learning outcomes and n subdivisions. The set Q = {1,2,3} represents the main qualifications defined by National Qualifications Framework (NQF) namely as: knowledge, skills and competence, which is standard regardless of the field of education (social sciences, business and administrative sciences, engineering, medicine etc.). Let:

W_j : Contribution value of a course to learning outcome j as identified by the department where j = 1, 2,..., о (see Table 2 for contribution values of some courses in HUBAP)

q^is6sub j^ : Relationship between outcome j and qualification t in terms of sub-article s where j = 1, 2, ...,o and t = 1,2,3 and s is dynamic depending on the field (see Table 3 for NQF outcomes of business and administrative sciences field in Turkey and learning outcomes of HUBAP ).

The Contribution Score of a compulsory course i in qualification t where t E Q is calculated as follows (see Table 4 for scores of one compulsory course in HUBAP):

... (1)

The Contribution Score of an optional course i in qualification where t E Q is calculated as follows:

... (2)

The Contribution Scores for compulsory and optional courses of subdivision i with m compulsory and e optional courses in qualification t where t E Q is calculated as follows (see Table G for HUBAP data):

... (3)

... (4)

The Contribution Score for the compulsory and optional courses of the program with n subdivisions contributing in qualification t where t E Q (see Table G for HUBAP data):

... (5)

... (6)

Note that there may exist compulsory and optional courses which arc not associated with any of the subdivisions. They are offered by other departments and represented with COUTR and COUTO, respectively. These values are calculated with the same methodology provided above.

4.Application

This section aims to demonstrate how the Bologna Process data of the programs obtained as in Section 3 can be utilized for academic program quality assessment and benchmarking. The methodology for assessment is selected as Data Envelopment Analysis (DEA) for its ability to handle multiple outputs and its capability of providing benchmarks for units evaluated. Below, we provide basics of DEA modelling and our model design.

4.1. Data Envelopment Analysis (DEA)

Data Envelopment Analysis (DEA) is a non-parametrie performanee measurement approach for identifying relative efficiency of Decision Making Units (DMUs) that are producing multiple outputs using multiple inputs. DEA has been presented to the literature by the study of Chames et al. [5]. Since then, DEA models have been widely applied for the real world organizations in different industries including public and private sectors all over the world, fn DEA modelling. The efficiency of a DMU is measured relative to all other units with the simple restriction that all DMUs lie on or below an efficient frontier. For each unit in a data set, a separate linear programming model is solved to investigate if there is a possibility for a unit to improve its performance. If there is no potential improvement for a unit (which means that it is performing efficiently relative to others), the linear programming model results in assigning an efficiency score of 100% to that unit. The unit(s) with 100% efficiency defines the efficient frontier. Below, wc provide basic modelling idea of DEA. For further information, see [8].

Let us consider n decision making units. Wc assume that each decision making unit j for j = 1,2uses m different inputs. Хц. For i = 1,2, ...,m and produces s different outputs.yrj. For r = 1.2, ...,s. The original DEA model proposed by Chames ct ah [5] is called as CCR model. Let ф represent, the efficiency score for unit o. Variables X¡ are introduced corresponding to each decision making unit (j = 1,2, ...,n) to form a Production Possibility Set (PPS) consisting of observed units, their convex combinations, scaled units (because constant returns to scale is assumed) and outperformed units. The linear programming formulation to calculate the efficiencv score of unit o is given below:

... (7)

s. t.

...

...

...

DEA is a powerful tool not only to calculate relative efficiency scores but also to identify benchmark units for inefficient units. Benchmark units can be identified through optimal Å values and they arc often referred as reference sets for inefficient DMUs. The reference (or benchmark) units represent radially the closest units on the frontier to the evaluated inefficient unit. The inefficient unit can adjust itself by making use of practices in the benchmark units. One of the major areas that DEA has been applied is higher education. It has proven itself as an effective tool to assess higher education performance and to derive promising policy implications (see Johnes [17] for a review). Different focuses on efficiency in studies applying DEA can be observed as teaching efficiency [1], research efficiency ([6], [16]) or overall efficiency where both research and teaching dimensions are considered ([19]). DEA is also used as a ranking tool to support student university selection ([4]).

4.2.Modelling of the Problem

We utilize the data organized for Business programs in Turkey for the program quality assessment and we approach this problem as an output-maximization type DEA problem. Figure 1 depicts where the intended evaluation stands in the higher education context. Of the two main functions of academic departments, academic program design is one of the key elements of teaching component. This is the part to which the Bologna Process is heavily contributing in European Higher Education context. It is possible to measure performance at any stage defined in Figure 1. Therefore, it is possible to define different factors at any stage evaluations. Wc primarily focus on the quality in program design and while doing that we make use of organized Bologna Process declarations of the academic programs. As mentioned above, DEA is an efficiency measurement tool, in which units arc relatively evaluated. Here, it is necessary to point out that wc borrow the idea of "efficiency" to serve as an indicator of academic program "quality" within the context of the Bologna Process declarations of the programs. Wc built our idea upon principles of the Bologna Process Quality Assurance System, which emphasizes continuous evaluation and improvement throughout the process. From this aspect, we establish an analogy between "evaluation" objective of Quality Assurance System and well-known quantitative "performance evaluation" stream in the Operational Research literature. Wc define the problem as a "performance evaluation" problem and propose that DEA can be a tool to measure the academic program quality in the Bologna Process declarations context. As a natural result of this analogy, the concept of efficiency becomes an indicator of academic program Bologna declaration quality

The sample consists of 38 Business undergraduate degree programs in Turkish Universities with the Bologna Process data available. The organization methodology in Section 3 is applied to all courses of 38 business programs and aggregate measures for subdivisions in each Business program are obtained. The data collection has required analysing learning outcomes for almost 4000 courses. Resulting data set is peculiar in two aspects. First, the determination of the input-output factors is exceptional. Second, the values of factors are between 1 and 5.

DEA requires the identification of input and output factors. General tendency is that if the value of a factor is aimed to be maximized (minimized), then it is an output (input) factor. In that sense, we have a unique case that all factors obtained in preliminary analysis (contributions of the programs or subdivisions to qualifications) behave as outputs. Since there is no direct input to be identified to serve to produce those outputs, wc prefer to utilize fixed input DEA model where only input variable is taken as 1 for all units and focus on output maximization. Such model is known as 'pure output model' introduced by Lovell and Pastor [23]. Two relevant options might be considered as input factors here. One relevant option could be the number of courses in the given program. However, using such an input factor is debatable for a clear reason that the programs with richer sets of courses will be seemed as consuming more resources than others. Simply, the objective of an academic program is not to contribute more with less number of courses. Second option could be the number of faculty as in other DEA applications to higher education. However, it would not be accurate to assert that the contributions of programs to the knowledge, skills and competence arc directly associated with the quantity of people pursuing the program. Also, not all staff is fully engaged with teaching and it is impossible to differentiate since the numbers in the Higher Education Council are aggregate.

Another exceptional side of the data is associated with the output factor values. It is important to mention that in measuring educational quality, it is likely to eomc up with discrete factor values (such as satisfaction or ranking) and convexity assumption of DEA may not hold. However, for our ease, although the initial measures consist of discrete data points between 1 and 5, the final measures derived after data organization to be used in DEA are continuous (see Table 6). Therefore, convexity assumption is not violated. On the other hand, although the measures are continuous, there is another aspect to be considered that our output data are limited between 1 and 5, which requires handling. As mentioned in Section 4.1 envelopment form of DEA modelling is based on a production possibility set including sealed units because constant returns to scale (proportionality) is assumed. The proportionality assumption leads the sealed units to be members of the EPS. However, in our ease, sealed units with the output values less than 1 and more than 5 are infeasible. Thus, the PPS should be bounded with the units with output values less than or equal to 5. For that specific purpose, we design our DEA model based on Output Oriented Bounded Variable DEA modelling proposed by Cooper ct al. [7]. It enables setting upper and lower limits to output values in the production possibility set by simply incorporating following constraints to model (7), where l7r and Uį! represent lower and upper bounds for output values. For our ease, lĶ = 1 and = 5.

... (8)

... (9)

After the DEA model is specified as above, efficiency analysis of business programs is performed at two levels:

Level 1. Program-based Efficiency: At this level, the business programs in 38 universities of Turkey arc assessed based on their overall program contribution scores to the knowledge, skills and competence factors calculated regarding compulsory (CPROR) and optional (CPROO) courses (see highlighted rows of Tabic 6 for the data of HUBAP). There arc 6 outputs for each program (knowledge, skills and competence for compulsory and optional courses). Note that CPROR and CPROO scores are not exact averages of subdivision contributions (CDIVR or CDIVO) due to the fact that the contribution of courses offered by outside department (COUTR and COUTO) are also accounted in calculating CPROR and CPROO.

Level 2. Subdivision-based Efficiency: The efficiency is assessed at a sub-level. CDIVR and CDIVO scores of all subdivisions are used. Each of 5 subdivisions (Management & Organization, Finance & Accounting, Marketing & Production, Quantitative Methods and Business Law) under the Business Departments arc evaluated relative to analogous subdivisions in other departments. Therefore, the analysis consists of 5 different analyses conducted for each subdivision (sec Table 6 for the data of HUBAP subdivisions).

5.Results

Below, wc discuss some key findings of our assessment in business programs by 38 Bologna Process accredited departments in Turkey.

Efficient Programs and Reference Sets. The results for level 1 analysis reveal that the average effieieney score is 76.9% and 6 out of 38 programs are found to be efficient. The results for level 2 analysis reveal that there is no program efficient, in terms of all five subdivision contributions. Some programs are efficient in terms of 4 subdivisions (U36) or 3 subdivisions (U2, U3 and U23). These results reveal that being efficient in program-based evaluation docs not ensure efficiency in terms of all subdivisions. This fact points out the importance of the subdivision based analysis in order to put strengths or the weaknesses of the programs. An important point to note about level 1 and level 2 analyses is that the efficiency scores of the program as a whole at level 1 are not averages of efficiency scores of subdivisions at level 2 since level 1 data set includes additional contribution data from out-of-department courses. The analysis considers 6 different data sets analyzed separately. Table 7 summarizes average efficiency scores and number of efficient units for those 6 analyses. The highest average efficiency score level is attained by Marketing fc Production subdivisions of the departments.

For inefficient programs, Table 8 summarizes the count data for the reference set (benchmarks) both at program and subdivision levels that an efficient program found to be benchmark to others with respect to each analysis. For instance, in terms of program-based efficiency, program U2 is in the reference set of 20 programs. When Tables 7 and 8 arc compared, it is observed that some programs arc efficient but not benchmarks to any of others. This is observed in two subdivisions: Quantitative Methods (QM) and Marketing & Production (M&P). The number of efficient units in QM subdivisions is 9. Seven out of efficient units are found as benchmarks to others according to Table 8. It means that two subdivisions in QM are outliers.

Efficiency based on Different Categories of Universities. We collate the efficiency scores with respect to two perspectives. First, we consider a public-private university categorization. 30 out of 38 business programs in our data set are offered by public universities. Table 9 summarizes the average efficiency scores and number of efficient units regarding this categorization. Results reveal that programs offered in private universities attain higher efficiency levels than public ones. It is observed that among the listed reference programs in Tabic 8, four programs arc held by private universities. This reveals that half of tlie private programs in our sample arc found to be efficient in at least one of the models.

The second categorization relies on the age of the programs as an indication of experience. We classify the programs to three categories as long-established (est. before 1992), mid-aged (est. between 1992 and 2006) and recently-established (est. after 2006). Years 1992 and 2006 are deliberately selected. Both are progressive years in Turkish higher education when an expansion in number of new university establishment is observed. The number of universities in Turkey has increased from 29 to 53 during 1992. Another boost has begun in 2006. Between 2006 and 2011, 88 new universities have been established [11]. We have 7 longestablished, 13 mid-aged and 18 recently-established business programs in our data set. Lower panel of Table 9 presents the average efficiency scores for three different categories. In general, long-established universities attain higher levels of average efficiency. It may point out that experience has an effect on preparation of the Bologna Process information packages.

6.Critical Review of the Data and the Methodology

The Qualifications Framework of the Bologna Process aims at a standardized structure for Higher Education programs in Europe beginning from the course level at the bottom. For this purpose, universities in member states arc subjected to provide two types of crucial information for each academic program. One is about how much the courses in the program contribute to the learning outcomes of the program. Second is how learning outcomes are related with the National Qualifications Framework (see Tables 2 and 3 for examples). It is important to mention that these initial declarations of academic departments about their programs may involve a high level of subjectivity. The measures are in a 1 to 5 discrete scale and of course arc affected by the judgements of the responsible authorities (they may include the instructors of the courses, head of divisions or heads of departments). Sucii a subjective data collection process is not without its drawbacks. Despite its subjectivity and even reliability issues, Qualifications Framework together with the data generated in the scope of it is one of the key components of the Bologna Process adaptations of the universities. It is a large scale accreditation process and effects of subjectivity is inevitable. Analytical evaluation mechanisms are necessary to overcome its drawbacks. The generation of such data sets has been consuming various university resources in terms of both labor and funding. Therefore, the data is valuable, requires attention and surely requires auditing. These arc the exact, reasons behind the data organizations and further analytical evaluations put forth in this paper. One of the major findings is related to the lack of protective mechanism for over assignment of the contribution scores. Below, we critically evaluate this potential weakness of the system followed by a potential pre-emptive mechanism to avoid it. Any quantitative analysis targeting the Bologna Process declarations of the academic programs inevitably would call the following question to mind: "What if a department sets the contribution values of all courses in a program to all its learning outcomes as 5, which is the highest value possible?" Any normalization, including proposed one in this paper, will set the contribution score of such programs to national qualifications as 5. Such an extreme condition is not observed in our sample, however, currently, there is no pre-emptive mechanism for doing so. Alternatively, we can think of overrating. Since everything relies on what the responsible authority declares, it may not be possible to detect overrating in the current setting. This can be thought as the major drawback of the proposed methodology. Extremely rated programs will attain higher contribution scores, correspondingly, higher efficiency levels. However, the mentioned drawback is directly related with the nature of the Bologna Process framework itself. It is quite hard to impose control over misjudgement or overvaluation in such data generation process. Therefore, this can be seen not only a limitation of the proposed methodology but also as a limitation of the whole system.

To illustrate, assume two hypothetical courses (X and Y) in a program with 10 learning outcomes. The pcrson(s) in charge of Course X overvalues the course and assigns the contribution values of this course to 5 for all learning outcomes. This will lead contribution scores of Course X to national qualifications to be set as 5. On the other hand, Course Y is evaluated and its contribution values are assigned to each learning outcome as in Table 10. Its contribution scores will clearly be found as a value less than 5. Too many courses evaluated with extreme values as Course X in a program will cause the very high levels of scores for the program as a whole. Obviously, programs with moderately assigned contribution values will possess a disadvantage relative to programs with too many Course X type courses. If an upper limit for the assignment of contribution values had been defined, such a discrepancy between courses would be avoided. Suppose there is an upper limit of 35 in total for contribution values. In this case, Course Y would be within limits as its current form. However, the contribution values of Course X cannot be assigned as 5 now. Let us assume an extreme case that they insist on assigning 5s. (see course X' in Table 10). In this ease, only 6 values could be assigned as 5 and the rest must be 1. This will affect contribution score calculations for Course X (now it cannot, be 5 in average) and therefore, the disadvantage of Course Y would be eliminated. Such an adjustment would affect, the aggregation and lead up to more fair evaluations.

Although there exists a problem of potential extreme assignments, it does not devalue the importance of the Bologna Process declarations of the academic programs as indicators of program quality. It is also worth mentioning that DEA methodology to some extent overcomes this limitation and serves as a tool to detect outliers (i.e. potential extreme declarations). When examining the reference programs to inefficient ones, it is possible to observe that some programs can be found as efficient units but not as benchmark units to any of others. Such circumstances are also noticed in our empirical analysis (see Section 5). Those units are potential outliers and are located in the parts of the possibility set, which are hard to attain. For such units (programs), it can be wise to check for unrealistic declarations.

7.Conclusions

Program quality assurance is one of the core elements of the Bologna Process. The data of the academie programs collected in the scope of Bologna Process can serve as a rich resource for assessing program quality. This paper aims to provide an analytical approach for academic program quality assessment and benchmarking by utilizing the Bologna Process data of the academic programs. For this purpose, we propose, illustrate and critically review a methodology to come up with aggregate measures from the Bologna data so that program outcomes arc comparable. It is followed by using the obtained data for benchmarking purposes with Data Envelopment Analysis (DEA) at both program and subdivision levels. It is possible to identify various potential stakeholders who may benefit from of the proposed methodology. Although the clarifications of the methodology are focusing on a sample of business departments in Turkey, it is applicable to any type of academic program with the Bologna Process accreditation. The proposed data organization induces the contributions data to the commonly accepted qualifications framework that covers all types of academic programs in a member state and derives comparable measures enabling both university-wide and cross-university evaluations. DEA then serves as an effective tool to measure academic program quality and can provide beneficial information for higher education policy making. Potential stakeholders of the proposed methodology can be thought in three groups.

* Higher Education Councils and University Policy Makers. Being responsible of overall supervision of the Bologna Process in a member state, Higher Education Councils can benefit from current methodology in quality assurance, which is one of the main components of the Bologna Process sustainability. The university administrations from rectors to faculty deans arc main collaborators of Higher Education Councils for continuity of the process. Field-based overall analyses with the help of proposed standardization can produce an overlook on the Bologna Process adaptation of liable academic programs. It can be also beneficial to identify outliers (thus, the unrealistic bologna data declarations) so that corrective action can be taken on such academic programs. The nature of the method providing relative evaluations between academic programs can apparently produce information for determination of poorly designed programs and enable reformatory action through benchmarking. Therefore, the use of the proposed methodology can play a role in strategic planning of higher education within a certain institution or over the whole system.

* Departmental Authorities. The chairs of the departments, head of subdivisions and faculty members in a department can benefit in assessing and adjusting the academic program content and quality. Since the method establishes comparable measures and relative evaluations, it serves as a direct tool to observe an academic programs' position in terms of its contributions to commonly accepted qualifications among analogous programs or other programs held within the same university. A subdivision based analysis can put subdivision contributions to a specific program forward so that adjustments can be made in the content or distribution of the courses over subdivisions via benchmarking related programs or subdivisions. Thus, benchmarking information provided in the scope of the analysis plays an important role for the program adjustments of the departments.

* Students. The Bologna Process data for any accredited program is currently available online on related department web sites. However, in its current form, it is difficult to compare programs with each other. By providing aggregated measures of subdivision or program contributions to qualifications, the utilizing the proposed methodology as a quality assessment tool can also be effective for students to provide insight for university selection.

In conclusion, the paper contributes to the literature by providing a framework for organizing and utilizing the Bologna Process data for quality assessment of academic programs. We suggest that the use of analytical tools fits well with the quality assurance objective of the Bologna Process in the European Higher Education Area. The proposed methodology is adaptable to different evaluations and produces results that several stakeholders from policy makers at different levels to students can benefit to a great extent.

Sidebar

Received; August 10, 2017; accepted: October 16, 2018; available online: December 13, 2018

DOI: 10.17535/erorr.2018.0024

Footnote

* Corresponding author.

References

References

[1] Agasisti, T. and dal Bianco, A. (2009). Reforming the University Sector: Effects on Teaching Efficiency-Evidence from Italy. Higher Education 57(4): 477-498.

[2] Amaral, A. and Magalliacs, A. (2004). Epidemiology and the Bologna Saga. Higher Education 48, 79 100.

[3] Bornmann, L., Mittag, S., Daniel, H.D. (2006). Quality assurance in higher education metaevaluation of multi-stage evaluation procedures in Germany. Higher Education 52: 687-709.

[4] Bougnol, M.L and Dula, J.H. (2006). Validating DEA as a ranking tool: an application of DEA to assess performance in higher education. Annals of Operations Research 145: 339-365.

[5] Chames, A., Cooper, W.W., Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research 2: 429 444.

[6] Chcrchyc, L. and Abede, P.V. (2005). On research efficiency: a micro-analysis of Dutch university research in economics and business management. Research Policy 34(4): 495-516.

[7] Cooper, W.W., Seiford, L.M., Tone, K. (2000). Discretionary, non-diseretionary and categorical variables. In WAV. Cooper, L.M. Seiford and K. Tone (Eds.), Data Envelopment Analysis: A Comprehensive Text with Models, Applications and DEA-solvcr Software, (pp. 183 219). Kluwcr Academic Publishers, Boston.

[8] Cooper, WAV, Seiford LM, Tone К (2006). Introduction to Data Envelopment Analysis and Its Uses with DEA-Solver Software and References. New York, NY: Springer.

[9] European Higher Education Area Glossary http://www.ehea.info/eidl05808/glossary.html [Accessed 29 April 2017].

[10] European Higher Education Area History http://www.chca.info/pid34248/history.html [Accessed 29 April 2017].

[11] Gunay, D. and Gunay, A. (2011). Quantitative Developments in Turkish Higher Education since 1933. Journal of Higher Education and Science 1(1): 1-22.

[12] Gvaramadze, I. (2008). From Quality Assurance to Quality Enhancement in the European Higher Education Area. European Journal of Education 43(4): 443-445.

[13] Hacettepe University Bologna Information Package http://akts.haecttcpc.cdu.tr/ [Accessed 29 April 2017].

[14] Hang, G. (2003). Quality Assurance/Accreditation in the Emerging European Higher Education Area: a possible scenario for the future. European Journal of Education 38(3): 229-241.

[15] Huisman, J. and Westerheijden, D.F. (2010). Bologna and Quality Assurance Progress Made or Pulling the Wrong Cart? Quality in Higher Education 16(1): 63-66.

[16] Jotines, J. and Yu, L. (2008). Measuring the research performance of Chinese higher education institutions using data envelopment analysis. China Economie Review 19(4): 679 696.

[17] Jolinos, J. (2015). Operational Research in education. European Journal of Operational Research 243, 683 696.

[18] Kelim, B.M. (2010). Quality in European Higher Education: The Influence of the Bologna Process. Change: The Magazine of Higher Learning 43(3): 40-46.

[19] Kempkes, G. and Pohl, C. (2010). The efficiency of German universities-some evidence from nonparametrie and parametric methods. Applied Economies 42(16): 2063 2079.

[20] Kcttuncn, J. and Kantola, M. (2006). The implementation of the Bologna Process. Tertiary Education and Management, 12(3): 257 267.

[21] Kohler, J. (2003). Quality Assurance, Accreditation, and Recognition of Qualifications as Regulatory Mechanisms in the European Higher Education Area. Higher Education in Europe 28(3): 317-330.

[22] Kristensen, В. (2010). Has External Quality Assurance Actually Improved Quality in Higher Education Over the Course of 20 Years of the 'Quality Revolution'? Quality in Higher Education 16(2): 153 157.

[23] Lovell, C.A.K and Pastor, J.T. (1999). Radial DEA models without inputs or without outputs. European Journal of Operational Research 118, 46-51.

[24] National Qualifications Framework for Higher Education in Turkey (NQF-HETR.) http://tyye.yok.gov.tr/ [Accessed 29 April 2017].

[25] Pusztai, G. and Szabó, P.C. (2008). The Bologna Process as a trojan horse: Restructuring higher education in Hungary. European Education 40(2): 85 103.

[26] R.ozsnyai, C. (2003). Quality assurance before and after "Bologna" in the Central and Eastern region of the European Higher Education Area with a focus on Hungary, the Czech Republic and Poland. European Journal of Education 38(3): 271-280.

[27] Saarinen, T. (2005). Quality in the Bologna Process: from competitive edge to quality assurance techniques. European Journal of Education 40(2): 190 204.

[28] Stamoulas, A. (2006). Greece before the Bologna Process: Confronting or embracing quality assurance in Higher Education? Higher Education Policy 19, 433-445.

[29] The Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESC) (2015). http://www.enqa.eu/wp-content/uploads/2015/ll/ESG 2015.pdf [Accessed 29 April 2017].

[30] Van Der Wende, M.C. and Wcstcrhcijdcn, D.F. (2001). International aspects of quality assurance with a special focus on European Higher Education. Quality in Higher Education 7(3): 233 245.

[31] Vidal, J. (2003). Quality Assurance, Legal Reforms and the European Higher Education Area in Spain. European Journal of Education 38(3): 301-313.