A PRIMARY INVESTIGATION INTO PERFORMANCE MEASUREMENT SYSTEMS IN THE MALTESE FURTHER EDUCATION (FE) SECTOR

Abstract

This work investigates the presence of a performance measurement system in the small island state of Malta using data gathered in 2012. In this way the type of system and KPIs currently in use are delineated. An interpretative methodology was employed to gain a deeper understanding of performance measurement in FE organizations. Data were collected from seven FE vocational and academic organizations via interviews with top senior management team (SMT) officials and other internal and external stakeholders. Very few organizations have performance measurement systems. The use of KPIs is very narrow, usually focusing on pass rates. There is no external QA or performance measurement regime making it very difficult to compare different FE organizations. By utilizing qualitative data it is possible to uncover particular nuances in specific sectors in small states. Reliability is further enhanced since the number of organizations in any sector in a small state in usually low. This translates into a higher percentage of organizations in the study.

Headnote

Abstract. This work investigates the presence of a performance measurement system in the small island state of Malta using data gathered in 2012. In this way the type of system and KPIs currently in use are delineated. An interpretative methodology was employed to gain a deeper understanding of performance measurement in FE organizations. Data were collected from seven FE vocational and academic organizations via interviews with top senior management team (SMT) officials and other internal and external stakeholders. Very few organizations have performance measurement systems. The use of KPIs is very narrow, usually focusing on pass rates. There is no external QA or performance measurement regime making it very difficult to compare different FE organizations. By utilizing qualitative data it is possible to uncover particular nuances in specific sectors in small states. Reliability is further enhanced since the number of organizations in any sector in a small state in usually low. This translates into a higher percentage of organizations in the study.

Keywords: quality assurance; further education; qualitative data; performance measurement; KPIs; small island states; Malta

Introduction

This work aims to assess the performance measurement (PM) regime that currently exists in the Maltese further education sector (FE), excluding university.1) In 2009, the Maltese National Commission for Higher and Further Education (NCHFE) published a report, Further and Higher Education Strategy, 2020, whereby PM is perceived as a strategic objective in FE (NCHFE, 2009).2) In this report, PM is rationalized in terms of setting up a QA agency to monitor educational provision and ensure excellence in FE (ibid.) and may be conceived as a way whereby national objectives are aligned with broader EU objectives established by the Bologna Process (Crosier & Parveva, 2013). Besides, the right PM regime involving data systems may be used to conduct comprehensive analysis of FE productivity and workforce inclinations and movements (Wolf, 20113); Vandal, 20094)). This seems to indicate that some kind of feedback mechanism is necessary to ensure mission alignment. PM is one way whereby such feedback may be organized.

Even though currently Malta does not have a formal external PM system (Crosier & Parveva, 2013), this does not mean that no feedback mechanism(s) exists in the Maltese FE sector. Such mechanisms may be formal/informal or internal/external in nature. By utilizing a social constructionist framework, this paper demonstrates how different stakeholders (government, FE organizations, manufacturing, tourism, trade-union and university officials) perceive PM in FE to ensure policy alignment to state goals. This might offer a way whereby PM may be construed as a mosaic of differing meanings and may offer some reference if a QA agency is set up in the future.

This work commences with the background to the study, a literature review focusing on PM, quality assurance (QA) and key performance indicators (KPIs), research questions, methodology, results, and conclusion.

FE organizations in Malta

In this work the Maltese post-secondary educational sector (excluding university) will be referred to as the FE sector, following the definition stated by the NCHFE whereby FE is defined as 'all formal education of persons above the compulsory school age, leading to qualifications classified at NQF [National Qualifications Level] levels 1 to 5' (NCHFE, 2009). Table 1 delineates the nine major providers of FE in Malta, as listed on the NCHFE and Maltese Department of Education's websites.

FE organizations in Malta offer either vocational or A-level courses (Caruana, 2005; Edwards, 2005), the exception being SFA2 which offers a range of academic courses at both O- and A-level. Local vocational curricula are set by vocational organizations (i.e., VOCATIONAL 1 and 2) and verified by the National Qualifications Council (NQC), while academic curricula are set by the Matriculation and Secondary Certificate Board (Matsec). This contrasts with other countries where more variety exists in terms of educational provision. For example, in the UK and Australia, while some FE organizations focus on either vocational or A-level educational courses, other FE organizations form partnerships and offer both vocational and A-level courses (Muijs et al., 2006).

In all cases, FE curricula are dominated by the acquisition of skills and knowledge, thus supposedly following government educational policy (Edwards, 2005; Mifsud, 2005). For example, government policy for the Australian vocational sector has 'its roles of an industry skills trainer, the major adult education provider, a post-school education provider and a provider to school age students' (Keating, 2006), while in the USA, community colleges are perceived as drivers of economic growth in rural areas (Rephann, 2007).

Literature review

As beneficiaries of public funding, FE organizations must account for their actions and accomplishments to government and the wider society. More than ever they are required to exhibit wider benefits arising from their activities in line with value for money requirements. FE organizations, like other social institutions, are thus made accountable to a number of stakeholders 'through the delivery of improved public goods (Benneworth & Jongbloed, 2010)

In the search for applicable literature, PM, QA, and key performance indicator (KPI) studies in the higher education sector (HE) in the USA, UK, and Australia were consulted. This approach may be considered valid owing to the overlap that exists between the two sectors (FE and HE) and is supported by researchers who have followed such a strategy (Feather, 2012). The discussion starts with an analysis of feedback mechanisms, before focusing on PM. QA and KPIs are debated as part of, and as a consequence of, such PM systems.

A feedback mechanism may be necessary to ensure that specified educational aims are met by FE organizations. Feedback may be conceptualized as a single loop between outputs and pre-set goals (Pitkanen & Lukka, 2011) and is important, since it keeps track of certain indicators which may be used to correct policy problems in the short and long term, providing such measures are within the organization's control (van Riel, 2012). Failure to do so may lead to staff demotivation (ibid.). Delivering this information in a timely manner to the right people within an organization may lead to better decision-making if it is complete, methodical, objective, periodic, and trustworthy (Shannak, 2009; Parmenter, 2007; Hardie, 1998).

However, a Taylorist approach using hard data as PM indicators (e.g. financial results extracted from accounting statements and accounting management systems) has been criticized: it is seen as being historical, narrow in scope, based on a number of assumptions and estimates and having no strategic focus or basis for continuous improvement (Johnson & Kaplan, 1987). Such approaches are therefore 'tantamount to driving a car by the rear view mirror' (Niven, 2005).

Processes and goal quantification, as well as features and roles, are used to define PM.5,6) Some definitions include 'the process of quantifying the efficiency and effectiveness of action' (De Lima et al., 2008). However, the authors fail to mention what should be quantified. The mention of indicators and indicator type is prevalent in other definitions. Thus indicators may be 'inputs, processes of delivery of activities and services outputs and outcomes'.^ Other definitions are more precise, calling for 'systematic tracking' and the need for fixed targets5) as well as 'a special type of organizational surveillance aimed at revealing the extent to which an employee's performance diverges from managers' expectations' (Sewell et ah, 2012).

The feedback mechanism should thus engage staff, and be analytical and reflective, while being accurate and easily accessible (Wolf, 2011). Ideally, performance indicators from all stakeholders should be considered, but time and financial constraints may not allow this in smaller organizations (Kenny, 2005; Hardie, 1998).

PM in education is justified in light of QA and evaluation, as well as EU educational policy based on 'the strengthening of Europe as a knowledge economy as mandated by the Lisbon declaration and a single market' (Grek et ah, 2009). Furthermore, the right PM systems may be used to conduct comprehensive analysis of FE productivity and workforce inclinations and movements (Wolf, 2011; Vandal, 2009). Hence, PM may be evaluated within the general policy of the EU, that is, comparing, benchmarking, and indicators. These are not only linked to the knowledge economy but to EU policy on gender equity, standardized qualifications, and policy impact assessment (Atkinson et al., 2003). PM in education may be operationalized as quality assurance (QA) programmes.

Pragmatic definitions of QA include:

[the] systematic management and assessment procedures adopted by HE institutions and systems in order to monitor performance against objectives, and to ensure achievement of quality outputs and quality improvements' (Harman, 2000).

In education, a QA typology has been proposed (Table 2) (Harvey & Green, 1993) and seems to be seminal in nature (Law, 2010a; Lomas, 2007).

Perceptions of QA seem to gravitate towards the 'fitness of purpose' definition (Lomas, 2007; McNay, 2006). Studies in the UK illustrate that university academics perceive quality to be related to such a definition, to the detriment of quality enhancement, with many interviewees remarking on the loss of personal autonomy, the accent on processes rather than content, a drive towards conformity, the rise of managerialism, and a lack of flexibility (Lomas, 2007; McNay, 2006). Some lecturers did, however, note the advantages of QA - clear metrics, indicators which may be used for course planning, and cost efficiency (Lomas, 2007; McNay, 2006). The 'fitness of purpose' definition also ties with aligning students' needs and wants to the core process of education (learning situations).

QA is perceived as a student right (but not exclusively) on a worldwide basis (Akhter, 2008) and may be used by politicians keen to demonstrate that public funds are used effectively and efficiently (Da Costa, 2009; Zarkesh & Beas, 2004). The variety of QA definitions reflects the large number of stakeholders involved with terms like effectiveness, efficiency, equity, equality, and quality being used interchangeably (Akhter, 2008). In Ireland, for example, a common QA framework driven by the European Qualifications Framework (EQF) involves programme validation, communication, equality, staff recruitment, access, learner protection, learner assessment, and standard determination (Tierney & Clarke, 2007).

QA has been extensively criticized. First, different actors in FE may have different perceptions as to what QA is, its implementation, and its usefulness. QA may be perceived as taking into account the products and services offered as opposed to the knowledge, comprehension, and the bond between learners and lecturers (Jung, 2012). This may result in a distinction between QA as an outcome of alignment and alignment as a way of establishing QA. The meaning of quality is, therefore, ambiguous (Harvey & Green, 1993), contentious and controlled by stakeholder perceptions (Saarinen, 2008). Furthermore, there are other worrying trends emerging in QA: the quality of training in vocational organizations that is increasingly becoming subject to private interests (Snell & Hart, 2007), and uncritical policy borrowing (Kennedy, 2011).

There are three main ways whereby QA is achieved: total quality management (TQM) - now discredited as an quality instrument in education, performance indicators, which are very much in use around the world, and external quality monitoring (EQM) with a shift from quantitative to qualitative appraisals (Law, 2010). Incorporation of the UK FE sector has led to such an EQM regime, with middle managers being employed for this purpose (Briggs, 2005; Leader, 2004). EQM is thus associated with accountability, assessment, and audit (Elton, 1992), leading to a paradox, since QA is perceived as a tool strongly coupled to accountability, rather than effective teaching leading to less innovative teaching (Law, 2010; Lomas, 2007; McNay, 2006).

The discourse of QA in the UK is moulded on the precept of neoliberalism (Olssen et ah, 2004), accompanied by a shift from professionalism to performativity (Ball, 2003). While it is more likely that UK sixth forms are subject to industrial style QA procedures than was the case in the early nineties, such QA regimes are not deemed to be on a par with industrial QA systems (Stoten, 2012). This could be due to the variation found across such colleges, seemingly indicating that internal QA systems are still in a development phase (ibid.). To work efficiently, a QA system rests on choosing the right indicators for the organization concerned.

Such indicators are Key Performance Indicators (KPIs) (Parmenter, 2007). Such contemporary KPIs (Franco-Santos et al., 2012) may be defined in a number of ways, the key components being data measures which are forward-looking as well as being significant for the present and future success of an organization (Shannak, 2009; Parmenter, 2007). Such measures are perceived to be 'organizationally relevant and operationally complete' (Ives, 2007), and might have to be collected at regular intervals (Fitz-Gibbon, 1996). Such a system of measures may illustrate where an organization is in terms of strategic goals.8)

KPIs have six basic characteristics: they are non-financial in nature; they are measured frequently; they are acted on by the senior management team; staff should understand the KPI and the corrective action that needs to be done; KPIs tie responsibility to the individual or team; KPIs have a significant and positive impact (Parmenter, 2007). However, KPIs have certain drawbacks: for example, the fact that many community colleges have multiple missions means that some indicators may contradict each other (Cohen & Brawer, 2003), the increase in bureaucracy and disregarding those indicators which are difficult to measure (Ewell, 1999), not having the right mix of hard and soft measures (Klenowski, 2009), and wrong implementation (Da Costa, 2009).

KPIs may be used to hold organizations accountable for what they accomplish (Wolf, 2011). Since attendance and graduation rates are not deemed sufficiently vital as KPIs9) (Wolf, 2011), there may be the need for a P-20 (post-20-years-old) longitudinal data system with the ability to locate student progress through post-secondary education, with data able to flow bidirectionally through the system.9) This way, universities and employers would have the right information at hand to take those decisions which are deemed necessary. In the same way, information from universities and employers ought to find its way to post-secondary organizations to better prepare students.9) In Florida, such a system 'includes employment data, providing a body of data linking success in school to success in the workplace'.10) Different KPIs at different levels with harder measures may be used at higher organizational levels (e.g., student completion rates), whilst softer measures used at the lower rungs of the organization, e.g. students' perceptions of teaching quality (Law, 2010b).

In light of the above, KPIs that may be used in education differ. Some authors11,12) stress a wide range of KPIs at different levels (e.g. strategic plans; organizational mission; institutional goals; best management practices; employment rates; class size; examination pass rates; transfer rates). Other authors are disinclined to mention strategic plans and mission (Alfred et. al, 1999; Sallis, 1990). While the latter authors agree with the former on examination pass and employment rates, they also introduce such KPIs as student, academic, and employer feedback. Table 3 displays these authors' broad range of KPIs.

So far, the discussion has focused on formal feedback. Informal communication channels are also important, given that collaboration among different stakeholders in the FE arena is indispensable to manage multifaceted real world problems. Informal communication has four characteristics (Kroon, 1995): (1) it conveys information rapidly; (2) it is very selective, especially with regards to sensitive information; (3) usually occurs within normal working hours; (4) works together rather than against formal communication channels.

Such collaboration may lead to a shared vision and commitment and an enduring assurance to formal and informal communication channels (Brown et al., 2011). It also includes those between vocational organizations and industry regarding students' progress and provision of new courses.13) Informal feedback may affect formal feedback by colouring people's expectations, people in this case being external stakeholders on one hand, and FE organizations' SMTs and academic staff on the other (Farr, 1993). Informal feedback is usually set on the premise that an organization is an information environment, where individuals actively seek information relating to their performance at work (Hanser & Muchinsky, 1978). In the FE context this is somewhat more complicated, e.g. a lecturer at a tourism vocational organization may seek information regarding a student's performance by directly contacting the hotel's HR manager. In this case, it seems that such feedback is not only important for the student but also for the lecturer concerned, since the latter might want to make sure that the course material is up to date.

While the usefulness of such feedback is perceived to be important (Becker & Klimoski, 1989), one major problem with informal communication is its haphazardness, which may lead to 'poorer training and employment outcomes for employers and trainees alike'.13)

Methodology

The above literature review has led the author to ask the following research questions: (1) what types of quality assurance procedures are used to ensure that FE organizations are aligned to government policy? (2) how do government, FE organizations, union, business, tourism, and university stakeholders perceive the success of national quality assurance procedures to ensure policy alignment to state goals?

To answer the above questions an interpretative/qualitative approach was utilized. Traditionally interpretative research is distinguished from positivism since the former does not produce numerical outputs based on hypothesis testing (Philamore & Goodson, 2004). This conceptual shift means that in modern social science research interpretative approaches are viewed as a particular research strategy rather than a group of methods, as a critique of positivist investigations and that it may produce theory out of research (ibid.). The ontological premise of interpretative techniques is relativism while emphasising that reality is not objective but rather socially constructed. Thus the richness of information is predominant (Veal, 2006).

Even though interpretative approaches may be relativist in nature, norms should be in place to determine that interpretative research is reliable and valid (Carcery, 2009). However it seems that some interpretative research agendas are not amenable to triangulation. Credibility, transferability, dependability and the confirmability are regarded as the four hallmarks by which trustworthiness may be verified (Lincoln & Guba, 1985).

The interpretative approach has been criticized on a number of grounds such as being soft, non-scientific and ineffectual for policy formulation (Philamore & Goodson, 2004) since different answers may be elicited by a different researcher. Secondly, behaviour is habitual and reflection does not really form part of most people's behaviour (Morrison, 2007). An individual's account may also be incomplete since accounts are based on the individual's understanding of society. Such an understanding is derived from a person's enculturation within society (ibid.) meaning that a researcher may have to discern whether they s/he has managed to connect with the values and worldviews of the informants they are studying (Clark et. ah, 1998).

Methods

Primary data was compiled through face to face semi-structured interviews with thirteen key informants. Semi-structured interviewing was used, since it allowed the researcher and informants to be free in their questions and responses. In this sense it may be termed a 'qualitative survey' (Jansen, 2010). Questions were not rigidly ordered, thus promoting a certain depth and breadth, thereby permitting new concepts to surface (Dearnley, 2005). Transcripts were written in grammatically correct English to lessen informant distress (Kvale, 1996), and informants were given the option to answer in English or Maltese.

In the case of interviews, judgemental sampling was used (Carcery, 2009). Probability sampling was excluded, since the number of FE organizations and the number of top government officials working directly with the FE sector is small. Senior FE organizations' SMT officials chosen in this study were approached (nine). High ranking officials from the NCHE, the Malta Union of Teachers (MUT), the University of Malta, and senior managers of a leading manufacturing company and a five-star hotel were also approached.

Contingencies forced certain changes to the above plan. Permission to interview the SMT officials at PRIVATE 1 and 2, was not granted with any reason given. Thus data from such organizations is non-existent, even though they represent a very small percentage in terms of students and academic staff. At VOCATIONAL 2, the chosen informant decided to delegate the interview to another member of the SMT. To partly mitigate the effects of the above, the researcher decided to interview two senior managers from two five star hotels. These particular hotels are known for their strong links with VOCATIONAL 2. Such an approach led to the triangulation of these two respondents' answers. In all cases anonymity was achieved by using pseudonyms (Table 4).

Table 5 delineates the main interview questions. To elicit data for the research question above, FE organizations' SMT officials were asked to specify government PM and QA procedures for their organization. The NCHE official was asked whether such PM and QA procedures existed, how data (if any) was collected, and whether such data (if collected) was used to ensure government policy/organizational mission alignment. The MUT official was asked whether PM and QA procedures existed and how data (if any) was collected. Business and tourism informants were asked whether their sector pro- vided feedback to the relevant FE organizations and vice-versa. The university official was asked whether university offers feedback to academic and Church FE organizations (excluding university).

Data analysis

Interview transcripts were analyzed using constructionist thematic analysis which is a type of content analysis and is justified by its wide and useful usage in the areas of education, management, and public policy (Gubrium & Holstein, 2008). The variety of positions held within the constructionist viewpoint make it hard to define (Stam, 2001) and may lead to terminological discrepancies (Pemecky, 2012). Constructionism states that the world is understood by constructing how we comprehend it individually and with others (Dodge et ah, 2005), highlighting freedom and the ability to use discourse. Constructionism is thus more likely to accentuate the way that humans understand the world (epistemological emphasis) rather the world itself (ontological emphasis) (Pernecky, 2012). Power and language manipulation and its relation to stakeholder groups become important since reality is 'mediated and manipulated' (Manson, 2008). FE organizations' SMT officials may afford a very different picture of their organization when compared to government, tourism, business and union officials. There may also be differences between FE organizations' SMT officials and teachers within the same organization. In this work, a weaker version of constructionism was employed allowing for partial objectivity (Newton et ah, 2011).

Results

There were no national PM or QA procedures in the Maltese FE sector since there was "no legislation requiring institutions to conduct external evaluation and auditing" (Simon) even though vocational organizations were required to follow the Malta Qualifications Council's (MQC) guidelines. This does not mean that no other feedback existed between FE organizations and external stakeholders. Informants answered the research questions in a number of different ways as outlined below.

RQ1: What types of quality assurance procedures are used to ensure that FE organizations are aligned to government policy?

This question elicited a variety of responses which may be grouped under formal and informal themes.

VOCATIONAL 1, VOCATIONAL 2, and CA2 all employed formal QA procedures, but still displayed diversity in the way the procedures were operationalized. At VOCATIONAL 1, Noel emphasized that QA was "really quite rigorous" and involved students' continuous assessment. This continuous assessment consisted of

[f]our assignments and normally one of them is time constrained. Each of these assignments is given to a coordinator who verifies the assignment. This coordinator (also called a verifier) will get a sample, usually about 10%, of the corrected scripts. If something is wrong he/she will call in all the scripts and correct them. Then there is the external examiner who will again do checks etc. We also have an independent QA unit to check that all documentation is in place at each Institute. At the end of the day it is quite rigorous. I think this is quite particular for some vocational colleges.

For Diana (industry), QA was expressed differently,

[I]ndustry would like to know what is happening to graduates. Industry feels that there is the need of a tracer study considering that VOCATIONAL 1 has been operating for ten years. Have they grown in their career? Has VOCATIONAL 1 helped them? Are they stuck? This will help understand the performance of VOCATIONAL 1.

The above contrasts with QA operationalization at CA2. Michael stated,

[R]esults are important. The majority of our students pass and go to university. At the end of the scholastic year every second year student and their parents have to fill in a detailed questionnaire covering each and every subject, the teacher and the teaching environment. It's a mix of qual and quant. It takes ages to analyze. I can see patterns and I think students are being reflective. Being the last day of school students are honest. Teachers are then given the results but confidentiality is always maintained. I don't go into classrooms.

Both vocational organizations offer City and Guilds courses. John admitted that,

[T]he latter [City and Guilds] do come and check us. We have to fill paperwork and this is confirmed by an external verifier.

Furthermore, at VOCATIONAL 2 other formal feedback is obtained from hotels. John stated,

[W]e check on hotels to ensure that students are in line with their training program (which is given to hotels). The student must follow this program. We monitor this though a department set up for this. We also have another department in charge of international placements. Remember students have to work abroad for six months as part of their training. This department makes sure that students' experiences are in line with what we want. There is a lot of checking going on.

Rachel and Elaine pointed out the importance of the above but added more. Rachel stated,

[A]part from pass rate I would certainly look at the success of their students. I would find out the percentage of students who actually remain in tourism and also check how many of them are in managerial positions. That way we would find out whether students wanted to go to VOCATIONAL 2 in the first place.

On the other hand, Elaine concentrated on personality change,

[A]part from pass rates I would certainly advocate the use of personality change in students. Confidence and positive approach should be included apart from academic KPIs. This will mean using a more holistic approach.

The types of formal KPIs used at these organizations vary. While pass rates as a KPI was common to all organizations, other KPIs emerge. Thus, Michael stated,

[S]ecurity is a problem. Students find it easy to come and go. We are not situated in a nice area and there are dangers. Security is a major issue. With separated parents you have to be careful since some parents might not have the right to see their children. I think one day we will have an incident.

The issue of security was also mentioned by Sandro,

[T]his is still lacking. We have pushed the notion as well as practical provisions. The problem is money. It is an important KPI.

All state funded academic organizations and CA1 employed informal internal QA procedures. Antonia stated,

[W]e don't have any official indicators. Results are not the only indicator since we rely on feedback from students and teachers which I consider important. It is not formal however.

This informality was also present at SFA1. Nigel declared,

[W]e have no formal indicators. University does not impose any KPIs. It's the responsibility of the Principal to see that there are no problems in the college. There is informal feedback between me and subject and area coordinators and maintenance staff.

At SFA3, the situation was more complex since the principal inherited an organization he perceived as,

[w]here teachers did practically whatever they liked. Trying to put my foot down on small things puts you in bad light, even though I knew I was right. If I had a say in choosing teachers for my school I would definitely go for graduates with a master's degree since they are more specialised. A bachelor's degree is certainly not enough at this level. This is still not part of the quality assurance procedure. The one and only thing they look at is the pass rate. When they send me a teacher, the government follows procedures that do not fit with our needs as a post-secondary school.

The above contrasted with Robin's (university) perceptions who was adamant that the only KPIs in use were examination pass rates but asked whether such a situation should continue,

[I]n addition to pass rates, FE colleges may have to craft and use additional KPIs which are not subject related and deemed to be less important to students and to university. Some other abilities may compensate for A-levels. Shouldn't you give such students a chance? I might be wrong but why not give students a second chance?

RQ2: How do government, FE organizations, union, business, tourism, and university stakeholders perceive the success of national quality assurance procedures to ensure policy alignment to state goals?

Answers from respondents determined that national QA procedures were only applicable to vocational organizations.

SFA1, SFA2, SFA3, CA1, and CA2 fall under this category. The answers given were, at times, ambiguous and contradictory as exemplified by Michael who stated,

[G]overnment regulations specify that inspectors visit the senior school to see what is being taught and how it is being taught. However sixth form is left out of this. They only come to check student numbers. So I fall under them for that but not for teaching. It makes my life a bit easier since I don't have the same amount of bureaucracy present in senior school. It seems that education stops at 16 leaving us in a void. NCF (National Curriculum Framework) suggested that sixth form students study 6 A-levels in first year and then go on to choose three in second year. They didn't consult us on this. Did they look at how implementation would take place? We were never asked for our suggestions. There is a complete lack of communication.

Antonia perceived the lack of QA in a different light. She stated,

[T]here is talk that eventually they [government inspectors] will have school inspections for sixth forms. From what we heard, since there are no people capable of doing this job, we don't know what's going to happen. The problem goes higher.

Since Maltese vocational organizations are responsible for their own curriculum development, the MQC ensures that such curricula are in line with EU standards. Noel stated that this is important since such a regime creates

[a]n emphasis on strategic clarity, so that everyone knows what should be measured or managed and knowing what really counts so that mat- ters are not trivialised. MCAST has created an environment in which everyone can learn and improve.

The lack of a national QA system was not reflected in the ongoing work at the NCHE with Simon stating,

[A]part from the legal document in front of Parliament (i.e. QA) we intend to apply for funding to prepare FE and HE institutions for this QA legislation. As an organization we have to prepare ourselves of dealing with external QA. The implementation will not happen overnight. Thus capacity building has to be factored in and it will require time and resources. We will probably have to engage an external evaluator who has the experience. However, we have to manage the process well - a process which is important and delicate.

Discussion

The results reported above indicate a number of issues, namely, the nature of PM (formal/informal) and its relationship to QA and KPIs, the presence or absence of external feedback mechanisms, and the nature of internal feedback mechanisms.

PM mechanisms in the Maltese FE sector appear to indicate a divide between vocational and academic organizations. Vocational organizations seem to have a more rigid, formal, and ongoing regime and may be similar to what is found in Ireland, that is, a system motivated by the EQF focusing on programme validation and standard determination (Tierney & Clarke, 2007). However, other attributes such as staff recruitment, access, learner protection, and equality seem to be absent (Tierney & Clarke, 2007).

The variety of QA systems present in academic organizations is also comparable to that in the UK and may indicate that internal QA systems are still in the development phase (Stoten, 2012). The informality in some academic organizations (e.g., CA1), may prove beneficial since smaller organizations may be more organic in nature. This does not mean that informal feedback is unimportant to larger organizations. Since Malta is a small island state, such informality may be easier to establish (Farrugia, 2002). Furthermore, the links between vocational organizations and industry had been established over a long period of time.

The kinds of KP Is used were very few with the emphasis on examination pass rates. While the latter is deemed important14) (Zarkesh & Beas, 2004), other KPIs which are strategic in intent are left out. These include best management practice, institutional goal attainment, and employment rates14) (Zarkesh & Beas, 2004; Alfred et ah, 1999). CA2 seemed to be the only FE organization that sought feedback from parents and students, as outlined by Sallis (1990).

`However, one may question the usefulness of the kind of feedback derived at CA2. Many academics point out the importance of timeliness, that is, feedback must be forward looking rather than backward looking (Shannak, 2009; Parmenter, 2007). There is the danger that obtaining feedback at the end of the scholastic year has no effect on the students exiting the system and is a classic case of Niven's (2005) 'driving a car by the rear view mirror'. The greater financial restraints facing smaller FE organizations may make it impossible for such organizations to engage in ongoing feedback mechanisms (Kenny, 2005).

Furthermore, while QA and KPI regimes focus on the systematic collection of data, it may not only be difficult, but also worrying to compare different FE organizations. Such comparisons that lead to a league table of FE organizations as is the case in the UK may do more harm than good since it may foster a climate of competition rather than collaboration. Furthermore, the small size of the Maltese FE sector and its rigidity makes it very difficult for academic FE organizations to change focus (e.g. it would be almost impossible for Church FE organizations to provide vocational training). On the other hand, vocational FE organizations are somewhat more flexible and are now offering Bachelor's degrees in some areas. This is accomplished in different ways: ITS offers its students a chance to pursuing a degree in tourism at the University of Malta while MC AST offers Bachelors degrees without any input from university.

The introduction of a standardized QA regime in the Maltese FE sector may lead towards a standardizing of organizations, and corresponds to performativity (Ball, 2003). Such a society is based on ritualization and managerialism as organizations are held accountable for what they do by establishing internal QA mechanisms. In this way the focus of FE organizations may shift from students to standards.

Conclusion

There are a number of lessons to be derived from this study. First, the application of QA systems, PM and KP Is in education is fraught with difficulty due to the tension that is usually perceived when such systems are implemented. Secondly, if external PM systems are implemented, then it might be right to include all stakeholders. The data collected reveals a wide spectrum of opinions coupled with suspicion with regards to certain stakeholders (e.g. Matsec). Such suspicion can be helpfully lessened by basing stakeholder relationships the use of informal feedback. Lastly, the small size of Malta and the assumptions that everyone knows everyone and short bureaucratic distances does not necessarily translate into organizational systems and processes that function faultlessly.

Footnote

NOTES

1. Malta only has one publically funded university. International universities offer some programmes in Malta.

2. http://planipolis.iiep.unesco.org/upload/Malta/Malta Further Higher Educ ation Strategy 2020.pdf

3. https://www.gov.uk/govemment/uploads/svstem/uploads/attachment data/f ile/180504/DFE-00031-pdfhttps://www.gov.uk/govemment/uploads/svstem/uploads/attachment data/fi le/1805 04/DFE-00031-2011 .pdf

4. http://www.ecs.org/docs/RevvingEdEngine.pdf

5. US Government Accountability Office (2010 [online]). BPR glossa-ry of terms: available from: www.gao.gov

6. Tasmanian Department of Treasury and Finance (2010) [online] Perfor-mance information - glossary. Available from: http://www.treasury.tas.gov.au

7. Standing Committee on Evaluation and Accountability (2001) [online] Glossary. Available from: http://www.scoea.bc.ca/

8. http://www.strategydriven.com/documents/SD_OrgPerfMsrs_Align.pdf OrgPerfMsrs Align.pdf

9. http://www.achieve.org/files/50-state-2009.pdf

10. http://www.achieve.org/files/Achieve_PolicyBrief_Dec18v4.pdf PolicvBrief Decl8v4.pdf

11. http://files.eric.ed.gov/fulltext/ED482702.pdf

12. http://www.ecs.org/clearinghouse/22/86/2286.pdf

13. http://bilbv.unilinc.edu.au/webclient/StreamGate7folder id=0&dvs=1414 340755916~766&usePid1=true&usePid2=true

14. Education Commission of the States (2000). State funding for community: colleges: a 50 state survey. Denver: ECS, Center for Community College Policy.

References

REFERENCES

Akhter, Z. (2008). Quality assurance in secondary education programme of Bangladesh Open University: present status and challenges. Turkish J. Distance Education, 9(2), 1-11.

Alfred, R., McClenney, K., Hudgins, J. & Ewell, P. (1999). Core indicators of effectiveness for community colleges. Washington: Community Col-lege Press.

Atkinson, T., Cantillon, B., Marker, E. & Nolan, B. (2003). Social indicators: the EU and social inclusion. Oxford: Oxford University Press.

Ball, S.J. (2003). The teacher's soul and the terrors of performativity../. Educ. Policy, 18, 215-228.

Becker, T.E. & Klimoski, R.J. (1989). A field study of the relationship between the organizational feedback environment and performance. Personnel Psychology, 42, 343-358.

Benneworth, P., & Jongbloed, B. W. (2010). Who matters to universities: a stakeholder perspective on humanities, arts and social sciences valorization. Higher Education, 59, 567-588.

Brown, D.R., Brewster, C.D., Karides, M. & Lukas, L.A. (2011). The phenomenon of collaboration: a phenomenologic study of collaboration between family medicine and obstetrics and gynecology departments at an academic medical center. Qualitative Report, 16, 657-681.

Briggs, A.R.J. (2005). Middle managers in English further education colleges: understanding and modelling the role. Educ. Management Administration & Leadership, 33, 27-50.

Carcery, M. (2009). The research audit trail - enhancing trustworthiness in qualitative inquiry. Electron. J. Business Res. Methods, 7(1), 11-24.

Caruana, H. (2005). The role of G. Curmi Higher Secondary School in increasing the participation rate (pp. 29-36). In: Camilleri, R. (Ed.). The Lisbon objectives and Maltese education provision. Floriana: Ministry of Education, Youth and Employment.

Clark, M., Riley, M., Wilkie, E. & Wood, R.C. (1998). Researching and writing dissertations in hospitality and tourism. Bedford: Thompson.

Cohen, A.M. & Brawer, F.B. (2003). The American community college. San Francisco: Jossey-Bass.

Dodge, J., Ospina, S.M. & Foldy, E.G. (2005). Integrating rigor and relevance in public administration scholarship: the contribution of narrative inquiry. Public Administration Review, 65, 286-300.

Grek, S., Lawn, M., Lingard, B. & Varjo, J. (2009). North by northwest: quality assurance and evaluation processes in European education. J. Educ. Policy, 24, 121-133.

Crosier, D. & Parveva, T. (2013). The Bologna process: its impact in Europe and beyond. Paris: UNESCO.

Da Costa Marques, M.C. (2009). Key performance indicators in Portuguese public universities. Res. Higher Educ. J., 5, 43-55.

De Lima, E.P., da Costa, S.E.G. & Angelis, J. J. (2008). The strategic management of operations systems performance. Intern. J. Business Perform. Management, 10, 108-132.

Dearnley, C. (2005). A reflection on the use of semi-structured interviews. Nurse Researcher, 13(1), 19-31.

Edwards, F. (2005). MCAST - a community vocational college (pp. 13-19). In: Camilleri, R. (Ed.). The Lisbon objectives and Maltese education provision. Floriana: Ministry for Education, Youth and Employment.

Elton, L. (1992). Quality enhancement and academic professionalism. New Academic, 1(2), 3-5.

Ewell, P.T. (1999). Linking performance measures to resource allocation: exploring unmapped terrain. Quality in Higher Education, 5, 191-209.

Farr, J. L. (1993). Informal performance feedback: seeking and giving (pp. 163-180). In: Schuler, FL, Farr, J. & Smith. M. (Eds.). Personnel selection and assessment: individual and organizational perspectives. Hillsdale: Lawrence Erlbaum.

Farrugia, C.J. (2002). Rewards and tribulations of senior education managers in small states (pp. 13-24). In: Baldacchino, G. & Farrugia,C.J. (Eds.). Education planning and management in small states: concepts and experiences. Essex: Commonwealth Secretariat.

Feather, D. (2012). Oh to be a scholar - an HE in FE perspective. J. Further & Higher Educ., 36, 243-261.

Fitz-Gibbon, C.T. (1996). Monitoring education: indicators, quality and effectiveness. New York: Continuum.

Franco-Santos, M., Lucianetti, L. & Bourne, M. (2012). Contemporary performance measurement systems: a review of their consequences and a framework for research. Manag. Account. Res., 23, 79-119.

Gubrium, J.F. & Holstein, J.A. (2008). The constructionist mosaic (pp. 3-12). In: Gubrium, J.F. & Holstein, J.A. (Eds.). Handbook of constructionist research. Thousand Oaks: Sage.

Hanser, L.M. & Muchinsky, P.M. (1978). Work as an information environment. Organ. Behavour & Human Performance, 21, 244-256.

Hardie, B. (1998). Managing monitoring and evaluation (pp. 163-176). In: Middlewood, D. & Lumby, J. (Eds.). Strategic management in schools and colleges. Wiltshire: Sage.

Harman, G. (2000). Quality assurance in higher education. Bangkok: Ministry of University Affairs and UNESCO.

Harvey, L. & Green, D. (1993). Defining quality. Assessment & Evaluation in Higher Education, 18, 9-34.

Ives, N. (2007). Organizational performance measures. USA: Strategydriven.com.

Jansen, H. (2010). The logic of qualitative survey research and its position in the field of social research methods. Qualitative Social Res., 11(2), 1- 21.

Johnson, H.T. & Kaplan, R.S. (1987). Relevance lost: the rise andfall of management accounting. Boston: Harvard Business School Press.

Jung, I. (2012). Asian learners' percpetion of quality in distance education and gender differences. Intern. Rev. Res. Distance & Open Learning, 13(2), 1-25.

Keating, J. (2006). Post-school articulation in Australia: a case of unresolved tensions. J. Further & Higher Educ., 30, 59-74.

Kennedy, K.J. (2011). Conceptualizing quality improvement in higher education: policy, theory and practice for outcomes based learning in Hong Kong. J. Higher Education Policy & Management, 33, 205-218.

Kenny, G. (2005). Strategic planning and performance measurement: develop and measure a winning strategy. Amsterdam: Elsevier.

Klenowski, V. (2009). Public education matters: reclaiming public education for the common good in a global era. Australian Educ. Researcher, 36, 1-26.

Kroon, J. (1995). General management. Pretoria: Kagiso Tertiary.

Kvale, S. (1996). Interviews: an introduction to qualitative research interviewing. London: Sage.

Law, D.C.S. (2010a). Quality assurance in post-secondary education: some common approaches. Quality Assurance in Education, 18, 64-77.

Law, D.C.S. (2010b). Quality assurance in post-secondary education: the student experience. Quality Assurance in Education, 18, 250-270.

Leader, G. (2004). Further education middle managers: their contribution to the strategic decision-making process. Educational Management Administration & Leadership, 32, 67-79.

Lincoln, Y.S. & Guba, E.G. (1985). Naturalist inquiry. Thousand Oaks: Sage.

Lomas, L. (2007). Zen, motorcycle maintenance and quality in higher education. Quality Assurance in Education, 15, 402-412.

Manson, S.M. (2008). Does scale exist: an epistemological scale continuum for complex human-environment systems. Geoforum, 39, 776-788.

McNay, I. (2006). Beyond mass higher education. Maidenhead: Open University Press.

Mifsud, H. (2005). Vocational education - is the Institute of Tourism Studies rising to the challenge (pp. 21-27). In: Camilleri, R. (Ed.). The Lisbon objectives and Maltese education provision. Floriana: Ministry of Education, Youth and Employment.

Morrison, M. (2007). What do we mean by educational research (pp. 13-36). In: Briggs, A.R. J. & Coleman, M. (Eds.). Research methods in educational leadership and management. Thousand Oaks: Sage.

Muijs, D., Harris, A., Lumby, J., Morrison, M. & Sood, K. (2006). Leadership and leadership development in highly effective further education providers: is there a relationship. J. Further & Higher Educ., 30, 87106.

NCHE [National Commission for Higher Education]. (2009). Further and higher education strategy 2020. Valletta: National Commission for Higher Education.

Newton, T., Deetz, S. & Reed, M. (2011). Responses to social constructionism and critical realism in organization studies. Organization Studies, 32, 7-26.

Niven, P.R. (2005). Driving focus and alignment with the balanced scorecard. J. Quality & Participation, 28, 21-25.

Olssen, M., Codd, J.A. & O'Neil, A.-M. (2004). Education policy: globalization, citizenship and democracy. Gateshead: Sage.

Parmenter, D. (2007). Key performance indicators: developing, implementing and using winning KPIs. New York: John Wiley & Sons.

Pernecky, T. (2012). Constructionism: critical pointers for tourism studies. Annals Tourism Res., 39, 1116-1137.

Philamore, J. & Goodson, L. (2004). Progress in qualitative research in tourism: epistemology, ontology and methodology (pp. 3-29). In: Phila-355 more, J. & Goodson, L. (Eds.). Qualitative research in tourism. Cornwall: Routledge.

Pitkanen, H. & Lukka, K. (2011). Three dimensions of formal and informal feedback in management accounting. Manag. Account. Res., 22, 125- 137.

Rephann, T. J. (2007). Community college growth opportunities: untapped potential in America's heartland. Growth & Change, 38, 443-459.

Saarinen, T. (2008). Whose quality: social actors in the interface of transnational and national higher education policy. Discourse: Studies Cultural Politics of Education, 29(2), 179-193.

Sallis, E. (1990). Corporate planning in an FE college. Educ.Management & Administration, 18(2), 30-34.

Sewell, G., Barker, J.R. & Nyberg, D. (2012). Working under intensive surveillance: when does 'measuring everything that moves' become intolerable. Human Relations, 65, 189-215.

Shannak, R.O. (2009). Measuring knowledge management performance. Eur. J. Scientific Res., 35, 242-253.

Snell, D. & Hart, A. (2007). Vocational training in Australia: is there a link between attrition and quality. Education & Training, 49, 500-512.

Stam, H. (2001). Introduction: social construction and its critics. Theory & Psychology, 11, 291-296.

Stoten, D.W. (2012). Exploring quality assurance in sixth form colleges. Quality Assurance in Education, 20, 259-273.

Tierney, L. & Clarke, M. (2007). The European qualifications framework: challenges and implications in the Irish further education and training sector. Eur. J. Vocational Training, 42(3), 129-143.

Vandal, B. (2009). Revving the education engine: effectively aligning education, workforce and economic development policy. Cincinnati: Knowledge Works Foundation.

Van Riel., C.B.M. (2012). The alignment factor: leveraging the power of total stakeholder support. London: Routledge.

Veal, A. J. (2006). Research methods for leisure and tourism: a practical guide. Dorset: Prentice Hall.

Wolf, A. (2011). Review of vocational education - the Wolf report. London: The Stationery Office.

Zarkesh, M. & Beas, A.M. (2004). UCLA community college review: performance indicators and performance-based funding in community colleges. Community College Review, 31(4), 62-73.