ABSTRACT
This paper attempts to develop an understanding of performance appraisal (PA) and to identify inconsistencies in its purposes and uses through an analysis of previous studies. The paper explores the purposes and uses of PA in the literature and devises a scheme for classifying the practice of PA into four categories - administrative, developmental, strategic, and role definition. This work was done to expand our understanding of performance appraisal and to highlight its uses in connection with other human resource management and development functions. The study finds that previous researchers devoted the most attention to PA conducted for administrative purposes, followed by PA for developmental purposes; the strategic category of PA was studied somewhat, whereas PA for purposes of role definition was almost ignored. The author offers a full-scale inventory of the purposes and uses of PA for researchers, performance raters and ratees. The study proposes that raters need to focus on the broader dimensions of PA, not simply its administrative functions.
Keywords: performance appraisal, administrative, developmental, strategic, role definition
INTRODUCTION
Performance Appraisal (PA), as an important area of behavioural science research, constitutes the basis for HR practices and lays the very foundation for research-based innovations (Murphy & Cleveland, 1995). PA has been widely researched and continues to be practiced, formally or informally, in almost all kinds of organisations, including business schools (Solomon & Hoffman, 1991), business houses (Wiese & Buckley, 1998), government (Amba-Rao, Petrick, Gupta, & Von der Embse, 2004) and non-government organisations (Amos- Wilson, 1996). People who conduct PA vary in their perceptions of its proper purpose, intended outcomes and manner of implementation (Murphy & Cleveland, 1995). This is primarily because of differences in leniency/strictness bias, accuracy of judgment, structure and administrative practices. This lack of uniform understanding of the purposes and uses of PA leads to dissatisfaction with it, particularly in Asian organisations, in which the use of PA is already limited (Cheng & Cascio, 2009; Gratton, 2004). This paper reviews and analyses the literature on PA published during the past 50 years (1959-2009) with the objective of organising, for the first time, an inventory of its purposes and uses. The paper presents four categories - administrative, developmental, strategic and role definition - and assesses the weight given to each category in the literature. This inventory should help to improve the understanding of stakeholders (i.e., raters, ratees and organisations, particularly in Asia) about how to apply PA in different situations. Thus, the paper attempts to broaden our understanding of the purposes and uses of the PA system by bundling it with other human resource management and development practices.
Previous researchers have explored the purposes and uses of PA in different ways. Stewart and Stewart (1977) gave precedence to administrative uses, grouped into four categories: raters, ratees, central planning and control, and outside parties. Dorfman, Stephan and Loveland (1986) focused on two basic purposes of the performance appraisal process: administrative and developmental. Cleveland, Murphy and Williams (1989) presented a list of 20 uses with administrative focus, grouped into four categories - between-person, within-person, systems maintenance and documentation - and examined the impact of these uses on organisational decisions and actions. Aguinis (2009) proposed communication and organisational maintenance as sub-categories within the set of administrative purposes. Noe, Hollenbeck, Gerhart and Wright (2003) added strategic purposes, and Youngcourt, Leiva and Jones (2007) added role definition to the categories of administrative and developmental PA that Dorfman et al. (1986) created. No study on the purposes and uses of PA has inventoried all its dimensions in one place. This paper attempts to bridge this gap in the literature.
The ultimate objective of carrying out this literature review is to help us understand how to measure the effectiveness of PA - a central topic in the PA theory and a crucial problem in the PA practice. The effectiveness of PA reflects the usefulness of the PA process for both the rater and the ratee (Walsh & Fisher, 2005). Over the years, PA practitioners have used varied styles and techniques. These practitioners are generally guided by the notion of "what is" rather than "how it should be." It is desirable to measure the effectiveness of this practice for employees and for organisations (also suggested by Murphy & Cleveland, 1995). Efforts to devise measurement standards for effectiveness of performance appraisal have been under way for 30 years (Jacobs, Kafry, & Zedeck, 1980; Dobbins, Cardy, & Platz-Vieno, 1990; Murphy & Cleveland, 1995; Cook & Crossman, 2004; Walsh & Fisher, 2005). A major contribution came from Jacobs et al. (1980), who suggested three categories of measurement standards for evaluating the PA system. The first category is "utilisation criteria", which address why PAs are conducted. The second category is "qualitative criteria", a set of rules and practices that help one judge the adequacy and usefulness of the PA mechanism for all involved in the process. The final category is "quantitative criteria", which focus on reliability and accuracy. The present study is likely to contribute to further refinement of the utilisation criteria proposed by Jacobs et al. (1980).
This paper is limited to analysis of available literature on various aspects of PA such as virtual networks (Golden, Barnes-Farrell, & Mascharka, 2009), total quality management (Soltani, 2003; Amba-Rao et al., 2004), teaching appraisals (Johnson & Shields, 2007), 360-degree feedback (Garavan, Morley, & Flynn, 1997; Brutus, Fleenor, & London, 1998; Cacioppe & Albrecht, 2000; McCarthy & Garavan, 2001) and reverse reviews (Redman & Mathews, 1995). This exercise produced an inventory of 98 purposes and uses of PA across 16 subcategories, which clustered under the areas of administrative (11 subcategories), developmental (2 subcategories), strategic (2 subcategories) and role definition (1 subcategory). This classification was done to assign an appropriate place to each of the above subcategories and answer the following research questions:
1. What does an inventory of the purposes and uses of PA consist of?
2. How should this inventory be utilised?
METHODOLOGY
The author performed domain analysis of the topic in order to devise a scheme to critically review the published literature, consistent with Ponis, Vagenas and Koronis (2009). The scheme was comprised of five steps: selection of appropriate search terms, search of established databases, initial filtering of the sample based on relevance, classification of purposes into groups and uses of PA and further filtering of the sample based on availability of material. The database search was performed using three search terms: "performance", "appraisal" and "rating". However, to narrow the search to the purposes and uses of PA, the terms "administrative", "developmental", "strategic", "role definition", "purpose" and "use" were used to search within the initial search results. Articles on PA were extracted from well-known databases (see Table 1). The search was conducted against the full text of articles as well as title, keywords, and abstract; it yielded 1,001 articles. For purposes of classifying articles into groups and filtering the sample based on availability and relevance, titles and abstracts were examined directly. This resulted in the selection of 114 articles, few of which pertained exclusively on purposes and uses of PA (e.g. Cleveland et al., 1989; Boswell & Boudreau, 2002; Youngcourt et al., 2007).
For purposes of selecting articles, the databases were explored sequentially. Arbitrarily, the author started with Emerald, followed by EBSCOhost, Taylor & Francis, JSTOR, SpringerLink, Blackwell Synergy, Palgrave Mcmillan and Walter De Gruyter. For certain articles, Questia, the online library, and Google Scholar, the online search engine for scholarly material, were also used. This scheme was intended to avoid duplication of articles. To ensure that the selected studies were reliable and valid for this review, the author focused on the search and selection of literature possessing three characteristics: relevance, quality and recency of the material.
Articles selected for the sample came from 64 journals. More than half came from five high-quality journals: Journal of European Industrial Training, The International Journal of Human Resource Management, Personnel Review, Journal of Management Development, and Journal of Managerial Psychology. The category of "others" comprised 45 journals, each contributing a single article to the sample (see Figure 1). Most of the sample articles were published in the 2000s (66) followed by the 1990s (35), 1980s (9), 1970s (3), and 1950s (1) (see Figure 2). In terms of the types of studies, 70 were research papers, followed by general reviews (16), viewpoint (12), literature reviews (10) and others (6) (see Figure 3). Figure 4 shows the first authors' country affiliations; 22 countries are represented. There were 14 articles by groups of researchers belonging to more than one country. Slightly more than half of the articles were by US- and UK- based researchers (46 and 26, respectively).
WHAT DOES INVENTORY OF THE PURPOSES AND USES OF PA CONSIST OF?
Administrative Purposes and Uses of PA
The most widely researched category of purposes and uses of PA was administrative, which constituted 59.23% of the literature in the sample. Other terms included in this category of purposes and uses were "judgmental" (Schweiger & Sumners, 1994), "evaluative" (Harrison & Goulding, 1997; Shelley, 1999), "personnel" (Poon, 2004; Turk, 2008), "summative" (Perillo, 2006) and "accountability" (Ovando & Ramirez Jr., 2007). The term "administrative", along with its substitutes, is frequently used to describe multiple purposes of PA. Among other things, PA conducted for administrative purposes is intended to evaluate performance of individuals and teams and to distinguish the ratee in comparison with others (Murphy & Cleveland, 1995; Thite, 2004). It is also used to assess the relative contribution of each employee in order to make sound administrative decisions about his or her salary, promotion and probation, and about lay-offs and other matters. (Schweiger & Sumners, 1994). High performance work systems value such administrative uses of PA especially for objective and equitable PA practices (Amba-Rao et al., 2004). This study finds that PA for administrative purposes helps managers make a variety of administrative decisions that strengthen the PA system, improve employee performance, ensure overall compliance with performance standards, develop HR systems, support HR activities, augment HR selection, conduct succession planning, manage employee relations, decide on compensation, encourage proactive approaches by the raters and incorporate the desired changes into the organisational climate. These dimensions are discussed in detail in the following paragraphs.
A strong PA system reduces the risk of lawsuits and losses in court (Schweiger & Sumners, 1994) by redressing employees' grievances (Fink & Longenecker, 1998) and by helping the rater and the ratee to address inaccurate perceptions about the purposes and uses of PA (Werther Jr. & Davis, 1996). The PA system is considered useful if it clarifies expectations and duties of the person being rated (Coates, 1994; Farmer, 2004) and requires the evaluator to discuss appraisal results with the employee (Patz, 1975; Havard, 2002). Discussion of the results persuades the ratee to accept the rater's findings and encourages both parties to support the PA system (Stewart & Stewart, 1977).
The core purpose of PA is to improve employee performance. PA results are used to review the employee's past performance (Roush, Curtis, Dershem, & Lovrich, 1991; Snape, Thompson, Yan, & Redman, 1998; Schraeder, Self, & Lindsay, 2006), resulting in the identification of poor performers (Jacobs et al., 1980) and recognition of good ones (Daley, 1991). Further, the process sets performance objectives for the next appraisal period (Daly & Kleiner, 1995; Leat & Lovell, 1997) based on the ratee's potential (Richards, 1959; Ammons & Rodriguez, 1986) and his or her relative worth within the department or organisation (Schweiger & Sumners, 1994). The PA helps set work targets (Bhattacharyya, 1999) and helps the employee look forward to improving his/her performance according to agreed-upon goals (Shaikh, 1995).
PA ensures overall compliance with performance standards and serves the organisation to develop merit criteria (Stephan & Dorfman, 1989) for the purpose of disciplining employees (Behery & Paton, 2008). It helps them understand codes of good practice (Stewart & Stewart, 1977), reducing legal liability. Documentation of PA also serves as a future reference, especially when an employee sues the company (De Cenzo & Robbins, 1996; Chow, 2004). Similarly, PA is used to establish lead and lag relationships to fulfill legal requirements within the organisation (Garavan et al., 1997; Timperley, 1998). PA results provide information to maintain equity among employees (Stewart & Stewart, 1977) and ensure compliance by the organisation.
PA results are used for updating personnel records (Farmer, 2004); revisiting job descriptions (Feild & Holley, 1975); deciding about transfers (Aldakhilallah & Parente, 2002; Shen, 2004; Islam & Rasad, 2006), implementing layoffs and recalls (Abu-Doleh & Weir, 2007), carrying out terminations (Edmonstone, 1996; Appelbaum & Grigore, 1997; Wells & Spinks, 1997; Whiting, Kline, & Sulsky, 2008), and handling probations (Behery & Paton, 2008). PA validate procedures used to promote good performers and demoting poor ones (Greguras et al., 2003; Pimpa, 2005; Roch, 2005) (Payne et al., 2009) and demoting poor ones (Boyd & Kyle, 2004). Thus, PA helps managers to evaluate the HR system by reinforcing the authority structure and management control (Cleveland et al., 1989; Soltani, Gennard, Van der Meer, & Williams, 2004a).
PA results help managers identify and diagnose job-design errors for purposes of correcting poor performance (Werther Jr. & Davis, 1996). These errors occur because of flaws in managing HR activities (Wilson & Western, 2000) such as HR planning (Deluca, 1993), forecasting, skills audits (Stewart & Stewart, 1977), and utilisation (Feild & Holley, 1975). If HR activities are poorly conceived, there is a likelihood of over- or under-staffing, lack of person-job fit, ineffective retention and poor employee engagement. These impediments can lead to faulty job design and, eventually, poor performance. Therefore, the ratee alone should not be held responsible for performance problems.
PA augments HR selection and helps facilitate evaluation of the job selection process (Shore et al., 1998). The review of HR selection processmakes the selection method more effective (Jacobs et al., 1980) by validating selection procedures, techniques and decisions (Ammons & Rodriguez, 1986; Walsh & Fisher, 2005; Jain, 2005).
PA provides a basis for succession planning by assessing future potential and promotability of the employee (Tziner, Murphy, Cleveland, & Roberts- Thompson, 2001; Myloni, Harzing, & Mirza, 2004; Chan, 2006). HR professionals use PA data to carry out succession planning activities such as placement (Beer, 1981), replacement, retention, discharge and tenure (Behery & Paton, 2008; Solomon & Hoffman, 1991). Moreover, by using PA for the above purposes, organisations can identify deficiencies in the staffing process; for example, good performance indicates strengths in recruitment and selection procedures and vice versa (Werther Jr. & Davis, 1996).
PA is used to provide information to manage employee relations. An efficient PA process offers supervisors and subordinates with opportunities to communicate with one another in performance planning and review meetings (Galang, 2004), which strengthens their working relationship (Shaikh, 1995) and improves the work environment (Wilson & Western, 2000).
PA facilitates decision-making about wages and salaries and about employee recognition and rewards (Smigel, 2000; Nickols, 2007). Such a compensation system motivates employees to improve performance (Law & Tam, 2008). PA engenders a pro-active approach among employees by diagnosing individual and organisational problems (Beer, 1981). The process points out the gap between "What is" and "What should be," providing long-term guidance to employees (Stewart & Stewart, 1977) as to how they should plan their work (Fink & Longenecker, 1998) and deal with internal and external challenges (Werther Jr. & Davis, 1996).
PA that complies with privacy restrictions can create a healthy "organisational climate" in which problems and grievances can easily be detected and handled (Stewart & Stewart, 1977). Such an organisational climate fosters progressive discipline that gives weight to the voice of employees (Galang, 2004; Lilley & Hinduja, 2006).
PA, when conducted for administrative purposes, is considered helpful in achieving favourable organisational outcomes such as employees' motivation, satisfaction and perceptions of fairness. In terms of the effectiveness of performance appraisal, the category of PA for administrative purposes performs five major functions. The first function is judgment. PA that is conducted for administrative purposes helps managers compare employees' performance. This results in fair treatment, which in turn leads to the second function: evaluation. The evaluation suggests improvements in the PA system at the individual, team, and organisational levels. The third function, accountability, helps assign responsibility for below-standard performances. The fourth function is summative assessment, which focuses on making the PA system effective. Once effective, the PA system is considered to be ready to contribute to the fifth function: personnel. This function enhances the utility of PA by using its results/data for decision-making in a variety of HR areas, as described under the 11 subcategories of administrative purposes and uses of PA.
Developmental Purposes and Uses of PA
The second most frequently researched category of purposes and uses for PA is developmental; the topic was covered in 26.73% of the reviewed literature. Some researchers use alternative terms for developmental such as "coaching" (Schweiger & Sumners, 1994), "motivational" (Caruth & Humphreys, 2008) and "formative" (Ovando & Ramirez Jr., 2007). These purposes and uses of PA are for evaluating performance while focusing solely on the employee's competence (Youngcourt et al., 2007). PA addresses developmental purposes and uses with methodologies such as: self appraisal (Baruch, 1999); upward appraisal (Adsit et al., 1994), which is also known as "bottom-up", "reverse review" and "upside- down" appraisal (Mathews & Redman, 1997); peer evaluation of individuals (Fisher, 1994) and teams (Thite, 2004); 360-degree feedback (Garavan et al., 1997); and multi-source feedback (Golden et al., 2009). These methodologies are used in addition to routine PA exercises (Redman & Mathews, 1995). Managers who are familiar with the use of PA for developmental purposes can devise mechanisms for improving performance through developmental activities at the organisational and individual level.
For development at the organisational level, PA results provide information based on which training and development (Soltani, Van der Meer, Gennard, & Williams, 2004c; Nurse, 2005; Islam & Rasad, 2006) and management development programmes are conducted (Cacioppe & Albrecht, 2000). Moreover, PA can help identify an industry's training requirements (Cleveland et al., 1989) along with employee development to manage change and organisational development (Ovando & Ramirez Jr., 2007).
There is strong evidence that the overall development of employees increases their levels of satisfaction and commitment (Wiese & Buckley, 1998). For development at the individual level, PA results provide employees with performance feedback (Reid & Levy, 1997; Blackmore, 2005; Narcisse & Harcourt, 2008), help define their career development needs (Hempel, 2001; Nickols, 2007) and determine their career paths (Spinks, Wells, & Meche, 2004; Law & Tam, 2008). PA generates a need for micro-training, coaching and counselling (Amba-Rao et al., 2004; Islam & Rasad, 2006; Law, 2007) to help employees meet their personal goals (Sinclair & Zairi, 1995). Thus, the provision of opportunities for self-development (Baruch, 1996) equips employees with new knowledge and skills (Odhiambo, 2005; Kuvaas, 2006). PA strongly relates to training. Various research studies endorse training needs assessment (TNA). Among 98 uses of PA, TNA is the most cited term, appearing in 8.85% of the reviewed literature (see for example DiLauro, 1979; Oppenheimer, 1982; Banks, Bures, & Champion, 1987; Sims, Veres, & Heninger, 1989; Noble, 1997; Al- Khayyat & Elgamal, 1997; Virmani, 2000; Heraty & Morley, 2000; Elbadri, 2001; Brown, 2002; Papalexandris & Chalikias, 2002). In addition, according to Behery and Paton (2008), PA provides a foundation for training evaluation.
If planned effectively, PA for developmental purposes can be helpful in designing a comprehensive framework for employee development. PA conducted for development purposes manages development at the individual level - through micro-training, coaching and career development - and at organisational level - through a focus on employee motivation, employee commitment etc. The development function enhances the view of PA because PA results and data can be used for decision-making in a variety of HR areas, as discussed under the two subcategories of developmental purposes and uses of PA.
Strategic Purposes and Uses of PA
PA conducted for "strategic" purposes was the subject of 10.96% of the literature reviewed. Strategic PA serves two major purposes. First, it establishes a functional relationship between the goals of organisation and the goals of its employees (Aguinis, 2009) by identifying these goals, setting them and achieving them. This kind of PA improves employees' perceptions of organisational goals because employees learn how to evaluate whether or not goals are being achieved (Wiese & Buckley, 1998; Soltani, 2003). PA for strategic purposes also provides information for organisational planning (Walsh & Fisher, 2005) that increases organisational effectiveness (Spinks et al., 2004), productivity (Herdlein, Kukemelk, & Türk, 2008) and organisational performance (Fifteen steps to a complete human resource program, 1997; Buckingham & Vosburgh, 2001). Second, PA conducted for strategic purposes helps the organisation deal with legal issues (Law, 2007). It encourages compliance with anti-discrimination laws - especially in cases of selection and promotion (Spinks et al., 2004) - and with requirements for equal-opportunity employment (Lacho, Stearns, & Whelan, 1991; Nickols, 2007).
The implementation of PA for strategic purposes is considered useful for the goal orientation function. This type of PA contributes to the effectiveness of performance appraisal by establishing a fit between individual and organisational objectives.
Role Definition Purposes and Uses of PA
The final category, PA conducted for purposes of role definition, comprised only 3.08% of the articles reviewed. PA for role definition clarifies the structure of position-focused appraisal, a topic that seems relatively overlooked. This sort of PA identifies job tasks that are no longer required or appraisal areas that extend beyond job requirements (Youngcourt et al., 2007). This kind of PA helps ratees understand their strengths and weaknesses relating to their roles and functions (Hanley & Nguyen, 2005; Law & Tam, 2008). Gaining this understanding develops employees' ability to support others in internalising the organisation's culture, norms and values, thereby establishing organisational identity, developing organisational commitment and creating a positive and healthy organisational climate (Shaikh, 1995). The ultimate result of this type of PA is improved role clarity (Pettijohn et al., 2001). The best utilisation of PA for role definition depends on adequate use of feedback that helps the ratee understand his or her role. This concludes the list of PA categories. A holistic view of these categories and their functions can be found in Figure 5.
HOW SHOULD THE INVENTORY OF PURPOSES AND USES OF PA BE USED?
This paper examined the frequency with which four major purposes and uses of PA are covered in various research studies (see Figure 6). The literature included in the sample gave significant coverage to PA for administrative purposes (59.23%) and PA for developmental (26.73%) purposes. The literature gave less attention to strategic PA (10.96%) and PA for purposes of role definition (3.08%). These results clearly show that there is a need to shift the focus toward PA for role definition and strategic purposes.
Within the category of PA that is conducted for administrative purposes, the subcategories most cited in the literature were as follows: development of HR systems (15.77%), with an emphasis on transfers, layoffs and recalls, terminations, probations and promotions; making decisions on compensation (15.19%), with a focus on administering employee rewards and recognition; succession planning (9.42%), with a focus on assessing employees' future potential. Meanwhile, the use of PA for inculcating proactive approaches among raters and ratees was a neglected area of the literature, appearing in 0.58% of the studies. Organisational climate is another subcategory of administrative uses that is rarely discussed, appearing in 1.15% of the studies. Establishing a mechanism that strengthens the appraisal system and supports other HR activities such as HR planning, forecasting, skills auditing and utilisation. These HR activities have received attention in 1.54% of the articles. Augmenting HR selection and ensuring overall compliance with performance standards appeared in 2.50% and 2.69% of the articles respectively. Improving employee performance and managing employee relations by taking management actions based on PA results appeared with reasonable frequency in the literature (5.38% and 3.46% respectively). As for the subcategories of PA for developmental purposes, development at the organisational level gained less attention in the literature sample (3.27%) compared with PA for development at the individual level (23.46%). Among studies that addressed the use of PA for individual development, feedback and TNA had significant shares (5.96% and 8.85% respectively). Among studies citing the use of PA for organisational development, training and development was the most studied use of PA (2.12%). Within the category of PA conducted for strategic purposes, the largest share (9.04%) came from studies describing the use of PA to accomplish organisational goals - their identification, setting, and achievement - while 1.92% was from studies of PA that targets legal issues. Among studies of PA that is used for role definition, the leading subcategory (1.35%) was PA that informs ratees of their strengths and weaknesses with respect to role and functions.
The literature indicates that performance appraisals often serve multiple purposes simultaneously (Cleveland et al., 1989). For example, while appraising employee performance for administrative purposes, an organisation can use the results of a single PA exercise to improve employee performance, augment HR selection and decide on compensation.
Can administrative, developmental, strategic and role definition purposes be accomplished from a single PA exercise? Before we answer this, it should be kept in mind that the simultaneous use of all categories in one PA exercise requires a higher degree of clarity among raters and ratees. The literature offers no consensus on the simultaneous use of PA for different purposes. For example, Youngcourt et al. (2007) maintain that the purposes of PA are distinct and need to be used separately. Cacioppe and Albrecht (2000) and Islam and Rasad (2006) contend that PA used for administrative purposes has immediate and tangible consequences, such as pay and promotion for the person being rated. PA conducted for developmental purposes, on the other hand, has no such consequences; it only affects the employee's learning and development. Thus, different purposes demand different approaches to PA.
For effectiveness of PA in terms of perceived fairness, the simultaneous use of all PA categories seems difficult to practice because there is a gap between an organisation's reasons for using PA and employees' perceptions about its use (Chang & Hahn, 2006). This gap is two-fold: administrative and developmental. The administrative gap occurs when ratees lose confidence and trust in raters if they perceive that PA is performed unfairly, especially when raters purposefully establish lead and lag relationships for administrative use (Goffin, Jelley, Powell, & Johnston, 2009). Raters are deemed as "foe" because ratees perceive the ratings to be unfavourable, even if they are judicious. The developmental gap occurs, meanwhile, when ratees show no concern about either their ratings or their raters. The employees perceive nothing to be at stake because raters are less likely to inflate or deflate ratings for the same reason. Raters are deemed as "friend" because ratees know that neither favourable nor unfavourable ratings have an impact on their pay or likelihood of promotion.
PRACTICAL IMPLICATIONS AND FUTURE RESEARCH
Managerial Implications
Human resource management and development are associated with the use of PA (Addison & Belfield, 2008). Therefore, the inventory developed by the author not only serves to improve our understanding of PA but also helps us integrate it with other HR activities such as HR forecasting, selection, skills audit and succession planning. The author's contribution shows clearly that the prevailing trend in previous research was to focus on the use of PA for administrative purposes followed by PA for developmental purposes. The review of the literature also reveals that the use of PA is growing rapidly (Soltani et al., 2004b). However, when it comes to a collectivist society like Pakistan, where it is hard to single out an individual who is accountable for results, a comprehensive and efficient evaluation system becomes difficult to implement. Therefore, PA as a managerial tool for maximising individual performance must be purpose-based (Chow, 2004).
Another implication of the inventory developed by the author is its use for the learning and development of raters. Employee perceptions of the usefulness of PA are affected by raters' training in the purposes of PA (Whiting & Kline, 2007). Such training would also serve as a useful tool to avoid inflated or deflated ratings.
Theoretical Implications
The application of the utilisation criteria for measurement of Effectiveness of Performance Appraisal (EPA) gains support from expectancy theory, which says that to raise employees' interest in the organisational setting, they should be rewarded according to their performance (Kermally, 2004). Similarly, social exchange theory explains that if, by virtue of the PA, an individual feels the organisation is keen on his or her development, he or she will then respond reciprocally (Youngcourt et al., 2007). Goal-setting theory maintains that ratees use performance feedback to evaluate their performance in comparison with their set goals (van Dierendonck, Haynes, Borrill, & Stride, 2007). Most of the literature focuses strictly on administrative and developmental uses of PA (see e.g. Dorfman et al., 1986); the current paper attempts to refine the concept further by adding two categories of use: PA for strategic purposes and PA for the purpose of role definition (Noe et al., 2003; Youngcourt et al., 2007).
Limitations and Future Research
This paper only aimed at expanding our understanding of "what" and "how" aspects of purposes and uses of PA. Therefore, segregation of context (i.e. countries, industry type, sectors, etc.) was avoided. While deciding on methods and materials, the author practiced the utmost care and collected material from selected databases to ensure a high standard of quality. However, due to lack of empirical evidence in the area under study, the author did not attempt to use search terms such as "a review of ...," "a meta-analytic review of ..." or "a narrative review of ..." etc. in the title. Instead, the author followed only those guidelines set down by Tranfield, Denyer and Smart (2003) and Armitage and Keeble-Allen (2008), which were useful and supported a methodology consistent with the available resources.
The objective of this paper was to develop an inventory of the purposes and uses of PA. However, apart from a word of caution about multiple uses of PA, empirical evidence has yet to be provided on "what is" and "what should be" within the practice of PA. To avoid the effect of nuisance variables, future research should identify and segregate types of PA according to methodology - e.g., self-appraisal, peer evaluation, multi-source feedback and reverse reviews - because the purpose for which PA is being used may influence the choice of methodology being used. This gap in our knowledge needs to be filled by expanding PA theory further. Research is also needed on the use of utilisation criteria for measuring effectiveness of PA. After Jacobs et al. (1980), no significant work on the subject has been published. Further research is needed on the dimensionality and construct validation of criteria for measuring the effectiveness of PA. Such research should build on the long-established approach to classifying PA by its purposes and uses, and approach that calls for broadening nomological networks in the field based on empirical evidence.
ACKNOWLEDGEMENT
The author is grateful to Dr. Aurangzeb Zulfiqar Khan and Dr. Sajid Hussain Awan for their insightful comments, which have been very useful in improving the paper.
REFERENCES
Abu-Doleh, J., & Weir, D. (2007). Dimensions of performance appraisal systems in Jordanian private and public organizations. The International Journal of Human Resource Management, 18(1), 75-84.
Addison, J. T., & Belfield, C. R. (2008). The determinants of performance appraisal systems: A note (do Brown and Heywood's results for Australia hold up for Britain?). British Journal of Industrial Relations, 46(3), 521-531.
Adsit, D. J., Crom, S., Jones, D., & London, M. (1994). Management performance from different perspectives: Do supervisors and subordinates agree? Journal of Managerial Psychology, 9(2), 22-29.
Aguinis, H. (2009). An expanded view of performance management. In J. W. Smither, & M. London (Eds.), Performance management: putting research into action (pp. 1-44). San Francisco: Jossey-Bass.
Aldakhilallah, K. A., & Parente, D. H. (2002). Redesigning a square peg: Total quality management performance appraisals. Total Quality Management & Business Excellence, 13(1), 39-51.
Al-Khayyat, R. M., & Elgamal, M. A. (1997). A macro model of training and development: validation. Journal of European Industrial Training, 21(3), 87- 101.
Amba-Rao, S. C., Petrick, J. A., Gupta, J. N. D., & Von der Embse, T. J. (2000). Challenges posed to performance management by TQM gurus: Contributions of individual employees versus systems-level features. The International Journal of Human Resource Management, 11(1), 60-89.
Ammons, D. N., & Rodriguez, A. (1986). Performance appraisal practices for upper management in city governments. Public Administration Review, 46(5), 460- 467.
Amos-Wilson, P. (1996). Management training in UK NGOs: A small survey. Journal of European Industrial Training, 20(1), 15-19.
Appelbaum, S. H., & Grigore, M. L. (1997). Organisational change and job redesign in integrated manufacturing: A macro-organisational to micro-organisational perspective. Journal of European Industrial Training, 21(2), 51-62.
Armitage, A., & Keeble-Allen, D. (2008). Undertaking a structured literature review or structuring a literature review: Tales from the field. The Electronic Journal of Business Research Methods, 6(2), 103-114.
Banks, M. C., Bures, A. L., & Champion, D. L. (1987). Decision making factors in small business: Training and development. Journal of Small Business Management, 25(1), 19-25.
Baruch, Y. (1996). Self performance appraisal vs direct-manager appraisal: A case of congruence. Journal of Managerial Psychology, 11(6), 50-65.
Baruch, Y. (1999). Integrated career systems for the 2000s. International Journal of Manpower, 20(7), 432-457.
Beer, M. (1981). Performance appraisal: dilemmas and possibilities. Organisational Dynamics, 9(3), 24-36.
Behery, M. H., & Paton, R. A. (2008). Performance appraisal-cultural fit: Organizational outcomes within the UAE. Education, Business and Society: Contemporary Middle Eastern Issues, 1(1), 34-49.
Bhattacharyya, D. K. (1999). Managing people. New Delhi, India: Excel Books.
Blackmore, J. A. (2005). A critical evaluation of peer review via teaching observation within higher education. International Journal of Educational Management, 19(3), 218-232.
Boswell, W. R., & Boudreau, J. W. (2002). Separating the developmental and evaluative performance appraisal uses. Journal of Business and Psychology, 16(3), 391- 412.
Boyd, N. M., & Kyle, K. (2004). Expanding the view of performance appraisal by introducing social justice concerns. Administrative Theory & Praxis, 26(3), 249-278.
Brown, J. (2002). Training needs assessment: A must for developing an effective training program. Public Personnel Management, 31(4), 569-578.
Brutus, S., Fleenor, J. W., & London, M. (1998). Does 360-degree feedback work in different industries? A between-industry comparison of the reliability and validity of multi-source performance ratings. Journal of Management Development, 17(3), 177-190.
Buckingham, M., & Vosburgh, R. M. (2001). The 21st century human resources function: It's the talent, stupid! Identifying and developing talent, one person at a time, becomes our defining challenge. Human Resource Planning, 24(4), 17-23.
Cacioppe, R., & Albrecht, S. (2000). Using 360° feedback and the integral model to develop leadership and management skills. Leadership & Organization Development Journal, 21(8), 390-404.
Caruth, D. L., & Humphreys, J. H. (2008). Performance appraisal: Essential characteristics for strategic control. Measuring Business Excellence, 12(3), 24- 32.
Chan, D. C. (2006). Core competencies and performance management in Canadian public libraries. Library Management, 27(3), 144-153.
Chang, E., & Hahn, J. (2006). Does pay-for-performance enhance perceived distributive justice for collectivistic employees? Personnel Review, 35(4), 397-412.
Cheng, K. H. C., & Cascio, W. (2009). Performance-appraisal beliefs of Chinese employees in Hong Kong and the Pearl River Delta. International Journal of Selection & Assessment, 17(3), 329-333.
Chow, I. H. (2004). The impact of institutional context on human resource management in three Chinese societies. Employee Relations, 26(6), 626-642.
Cleveland, J. N., Murphy, K. R., & Williams, R. E. (1989). Multiple uses of performance appraisal: Prevalence and correlates. Journal of Applied Psychology, 74(1), 130- 135.
Coates, G. (1994). Performance appraisal as icon: Oscar-winning performance or dressing to impress? The International Journal of Human Resource Management, 5(1), 167-191.
Cook, J., & Crossman, A. (2004). Satisfaction with performance appraisal systems: A study of role perceptions. Journal of Managerial Psychology, 19(5), 526-541.
Daley, D. (1991). Performance appraisal in local governments [Introduction]. Public Productivity & Management Review, 14(3), 249-252.
Daly, D., & Kleiner, B. H. (1995). How to motivate problem employees. Work Study, 44(2), 5-7.
De Cenzo, D. A., & Robbins, S. P. (1996). Human resources management (5th ed.). New York: John Wiley and Sons.
Deluca, M. J. (1993). Handbook of compensation management. New Jersey: Prentice Hall.
DiLauro, T. J. (1979). Training needs assessment: Current practices and new directions. Public Personnel Management, 8(6), 350-359.
Dobbins, G. H., Cardy, R. L., & Platz-Vieno, S. J. (1990). A contingency approach to appraisal satisfaction: an initial investigation of the joint effects of organizational variables and appraisal characteristics. Journal of Management, 16(3), 619-632.
Dorfman, P., Stephan, W., & Loveland, J. (1986). Performance appraisal behaviors: Supervisor perceptions and subordinate reactions. Personnel Psychology, 39(3), 579-597.
Edmonstone, J. (1996). Appraising the state of performance appraisal. Health Manpower Management, 22(6), 9-13.
Elbadri, A. N. A. (2001). Training practices of polish companies: An appraisal and agenda for improvement. Journal of European Industrial Training, 25(2/3/4), 69-79.
Farmer, J. (2004). How to get the most from performance reviews. PT: Magazine of Physical Therapy, 12(11), 46-49.
Feild, H., & Holley, W. (1975). Performance appraisal-an analysis of state-wide practices. Public Personnel Management, 4(3), 145-150.
Fifteen steps to a complete human resource program. (1997). Management Development Review, 10(1/2/3), 37-39.
Fink, L. S., & Longenecker, C. O. (1998). Training as a performance appraisal improvement strategy. Career Development International, 3(6), 243-251.
Fisher, C. M. (1994). The differences between appraisal schemes: Variation and acceptability - Part I. Personnel Review, 23(8), 33-48.
Galang, M. C. (2004). The transferability question: Comparing HRM practices in the Philippines with the US and Canada. The International Journal of Human Resource Management, 15(7), 1207-1233.
Garavan, T. N., Morley, M., & Flynn, M. (1997). 360 degree feedback: Its role in employee development. Journal of Management Development, 16(2), 134-147.
Goffin, R. D., Jelley, R. B., Powell, D. M., & Johnston, N. G. (2009). Taking advantage of social comparisons in performance appraisal: The relative percentile method. Human Resource Management, 48(2), 251-268.
Golden, T. D., Barnes-Farrell, J. L., & Mascharka, P. B. (2009). Implications of virtual management for subordinate performance appraisals: A pair of simulation studies. Journal of Applied Social Psychology, 39(7), 1589-1608.
Gratton, R. (2004). Teacher appraisal: A lesson on confusion over purpose. International Journal of Educational Management, 18(5), 292-296.
Greguras, G. J., Ford, J. M., & Brutus, S. (2003). Manager attention to multisource feedback. Journal of Management Development, 22(4), 345-361.
Hanley, G., & Nguyen, L. (2005). Right on the money: What do Australian unions think of performance-related pay? Employee Relations, 27(2), 141-159.
Harrison, K., & Goulding, A. (1997). Performance appraisal in public libraries. New Library World, 98(1138), 275-280.
Havard, B. (2002). Performance appraisals. New Delhi, India: Kogan Page.
Hempel, P. S. (2001). Differences between Chinese and Western managerial views of performance. Personnel Review, 30(2), 203-226.
Heraty, N. & Morley, M. J. (2000). Human resource development in Ireland: Organizational level evidence. Journal of European Industrial Training, 24(1), 21-33.
Herdlein, R., Kukemelk, H., & Türk, K. (2008). A survey of academic officers regarding performance appraisal in Estonian and American universities. Journal of Higher Education Policy and Management, 30(4), 387-399.
Islam, R., & Rasad, S. M. (2006). Employee performance evaluation by the AHP: A case study. Asia Pacific Management Review, 11(3), 163-176.
Jacobs, R., Kafry, D., & Zedeck, S. (1980). Expectations of behaviorally anchored rating scales. Personnel Psychology, 33(3), 595-640.
Jain, P. (2005). Strategic human resource development in public libraries in Botswana. Library Management, 26(6/7), 336-350.
Johnson, L., & Shields, J. (2007). Lessons from management-union partnership in teacher performance appraisal in the New South Wales public education system. The International Journal of Human Resource Management, 18(7), 1214-1227.
Kermally, S. (2004). Gurus on people management. London, UK: Thorogood Publishing Ltd.
Kuvaas, B. (2006). Performance appraisal satisfaction and employee outcomes: Mediating and moderating roles of work motivation. The International Journal of Human Resource Management, 17(3), 504-522.
Lacho, K. J., Stearns, G. K., & Whelan, R. K. (1991). Performance appraisal in local governments: A current update. Public Productivity & Management Review, 14(3), 281-296.
Law, D. R. (2007). Appraising performance appraisals: A critical look at an external control management technique. International Journal of Reality Therapy, 26(2), 18-25.
Law, R., & Tam, P. (2008). Employees' perceptions of performance appraisal: The case of an upscale hotel in Hong Kong. Journal of Human Resources in Hospitality & Tourism, 7(1), 25-43.
Leat, M. J., & Lovell, M. J. (1997). Training needs analysis: Weaknesses in the conventional approach. Journal of European Industrial Training, 21(4), 143- 153.
Lilley, D., & Hinduja, S. (2006). Officer evaluation in the community policing context. Policing: An International Journal of Police Strategies & Management, 29(1), 19-37.
Mathews, B. P., & Redman, T. (1997). The attitudes of service industry managers towards upward appraisal. Career Development International, 2(1), 46-53.
McCarthy, A. M., & Garavan, T. N. (2001). 360° feedback processes: Performance improvement and employee career development. Journal of European Industrial Training, 25(1), 5-32.
Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational and goal-based perspectives. Thousand Oaks, CA: Sage Publications.
Myloni, B., Harzing, A. K., & Mirza, H. (2004). Host country specific factors and the transfer of human resource management practices in multinational companies. International Journal of Manpower, 25(6), 518-534.
Narcisse, S., & Harcourt, M. (2008). Employee fairness perceptions of performance appraisal: A Saint Lucian case study. The International Journal of Human Resource Management, 19(6), 1152-1169.
Nickols, F. (2007). Performance appraisal: Weighed and found wanting in the balance. Journal for Quality & Participation, 30(1), 13-16.
Noble, C. (1997). The management of training in multinational corporations: Comparative case studies. Journal of European Industrial Training, 21(3), 102- 109.
Noe, R. A., Hollenbeck, J. R., Gerhart, B., & Wright, P. M. (2003). Human Resource Management. London, UK: McGraw Hill.
Nurse, L. (2005). Performance appraisal, employee development and organizational justice: Exploring the linkages. The International Journal of Human Resource Management, 16(7), 1176-1194.
Odhiambo, G. O. (2005). Teacher appraisal: The experiences of Kenyan secondary school teachers. Journal of Educational Administration, 43(4), 402-416.
Oppenheimer, R. J. (1982). An alternative approach to assessing management development needs. Training & Development Journal, 36(3), 72-76.
Ovando, M. N., & Ramirez Jr., A. (2007). Principals' instructional leadership within a teacher performance appraisal system: Enhancing students' academic success. Journal of Personnel Evaluation in Education, 20(1-2), 85-110.
Papalexandris, N., & Chalikias, J (2002). Changes in training, performance management and communication issues among Greek firms in the 1990s: Intercountry and intracountry comparisons. Journal of European Industrial Training, 26(7), 342- 352.
Patz, A. (1975). Performance appraisal: Useful but still resisted. Harvard Business Review, 53(3), 74-80.
Payne, S. C., Horner, M. T., Boswell, W. R., Schroeder, A. N., & Stine-Cheyne, K. J. (2009). Comparison of online and traditional performance appraisal systems, Journal of Managerial Psychology, 24(6), 526-544.
Perillo, S. (2006). Practice enhancement: Optimizing teaching performance in schools. International Journal of Educational Management, 20(5), 365-379.
Pettijohn, L. S., Parker, R. S., Pettijohn, C. E., & Kent, J. L. (2001). Performance appraisals: Usage, criteria and observations. Journal of Management Development, 20(9), 754-771.
Pimpa, N. (2005). Teacher performance appraisal in Thailand: Poison or panacea? Educational Research for Policy and Practice, 4(2-3), 115-127.
Ponis, S. T., Vagenas, G., & Koronis, E. (2009). Exploring the knowledge management landscape: A critical review of existing knowledge management frameworks. In D. Harorimana (Ed.), Cultural implications of knowledge sharing, management and transfer: Identifying competitive advantage (pp. 1-25). Hershey, PA: IGI Global.
Poon, J. M. L. (2004). Effects of performance appraisal politics on job satisfaction and turnover intention. Personnel Review, 33(3), 322-334.
Redman, T., & Mathews, B. P. (1995). Do corporate turkeys vote for Christmas? Managers' attitudes towards upward appraisal. Personnel Review, 24(7), 13-24.
Reid, P., & Levy, G. (1997). Subordinate appraisal of managers: A useful tool for the NHS? Health Manpower Management, 23(2), 68-72.
Richards, K. E. (1959). A new concept of performance appraisal. The Journal of Business, 32(3), 229-243.
Roch, S. G. (2005). An investigation of motivational factors influencing performance ratings: rating audience and incentive. Journal of Managerial Psychology, 20(8), 695-711.
Roush, D. L., Curtis, C., Dershem, H. A., & Lovrich, Jr., N. P. (1991). The development of behavior-based performance appraisal processes in smaller local governments: Lessons from a case study. Public Productivity & Management Review, 14(3), 267-279.
Schraeder, M., Self, D. R., & Lindsay, D. R. (2006). Performance appraisals as a selection criterion in downsizing: A comparison of rank-order and banding approaches. Managerial Law, 48(5), 479-494.
Schweiger, I., & Sumners, G. E. (1994). Optimizing the value of performance appraisals. Managerial Auditing Journal, 9(8), 3-7.
Shaikh, T. S. (1995). Appraising job performance - to be or not to be? An Asian dilemma. The International Journal of Career Management, 7(5), 13-18.
Shelley, S. (1999). Diversity of appraisal and performance-related pay practices in higher education. Personnel Review, 28(5/6), 439-454.
Shen, J. (2004). International performance appraisals: policies, practices and determinants in the case of Chinese multinational companies. International Journal of Manpower, 25(6), 547-563.
Shore, T. H., Adams, J. S., & Tashchian, A. (1998). Effects of self-appraisal information, appraisal purpose, and feedback target on performance appraisal ratings. Journal of Business and Psychology, 12(3), 283-298.
Sims, R. R., Veres, J. G., & Heninger, S. M. (1989). Training for competence. Public Personnel Management, 18(1), 101-107.
Sinclair, D., & Zairi, M. (1995). Effective process management through performance measurement: Part III - an integrated model of total quality-based performance measurement. Business Process Re-engineering & Management Journal, 1(3), 50-65.
Smigel, L. M. (2000). Basic training for new managers. Chicago: Lowell House.
Snape, E., Thompson, D., Yan, F. K., & Redman, T. (1998). Performance appraisal and culture: Practice and attitudes in Hong Kong and Great Britain. The International Journal of Human Resource Management, 9(5), 841-861.
Solomon, R. J. & Hoffman, R. C. (1991). Evaluating the teaching of business administration: A comparison of two methods. Journal of Education for Business, 66(6), 360-365.
Soltani, E. (2003). Towards a TQM-driven HR performance evaluation: An empirical study. Employee Relations, 25(4), 347-370.
Soltani, E., Gennard, J., Van Der Meer, R. B., & Williams, T. M. (2004a). HR performance evaluation in the context of TQM A review of the literature. International Journal of Quality & Reliability Management, 21(4), 377-396.
Soltani, E., Van Der Meer, R. B., & Williams, T. M. (2004b). Challenges posed to performance management by TQM gurus: Contributions of individual employees versus systems-level features. Total Quality Management, 15(8), 1069-1091.
Soltani, E., Van Der Meer, R. B., Gennard, J., & Williams, T. M. (2004c). Have TQM organizations adjusted their performance management (appraisal) systems? A study of UK-based TQM-driven organizations. The TQM Magazine, 16(6), 403- 417.
Spinks, N., Wells, B., & Meche, M. (1999). Appraising the appraisals: Computerized performance appraisal systems. Career Development International, 4(2), 94- 100.
Stephan, W., & Dorfman, P. (1989). Administrative and developmental functions in performance appraisals: Conflict or synergy? Basic & Applied Social Psychology, 10(1), 27-41.
Stewart, V., & Stewart, A. (1977). Practical performance appraisal. Westmead, UK: Gower Press.
Thite, M. (2004). Strategic positioning of HRM in knowledge-based organizations. The Learning Organization, 11(1), 28-44.
Timperley, H. S. (1998). Performance appraisal: Principals' perspectives and some implications. Journal of Educational Administration, 36(1), 44-58.
Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207-222.
Turk, K. (2008). Performance appraisal and the compensation of academic staff in the University of Tartu. Baltic Journal of Management, 3(1), 40-54.
Tziner, A., Murphy, K., Cleveland, J., & Roberts-Thompson, G. (2001). Relationships between attitudes toward organizations and performance appraisal systems and rating behaviour. International Journal of Selection & Assessment, 9(3), 226- 239.
van Dierendonck, D., Haynes, C., Borrill, C., & Stride, C. (2007). Effects of upward feedback on leadership behaviour toward subordinates, Journal of Management Development, 26(3), 228-238.
Virmani, B. R. (2000). Managing people in organizations: The challenges of change. New Delhi, India: Sage Publications.
Walsh, K., & Fisher, D. (2005). Action inquiry and performance appraisals: Tools for organizational learning and development. The Learning Organization, 12(1), 26-41.
Wells, B. & Spinks, N. (1997). Counselling employees: An applied communication skill. Career Development International, 2(2), 93-98.
Werther Jr., W. B., & Davis, K. (1996). Human resources and personnel management (5th ed.). New Delhi, India: McGraw Hill.
Whiting, H. J., & Kline, T. J. B. (2007). Testing a model of performance appraisal fit on attitudinal outcomes. The Psychologist-Manager Journal, 10(2), 127-148.
Whiting, H. J., Kline, T. J. B., & Sulsky, L. M. (2008). The performance appraisal congruency scale: An assessment of person-environment fit. International Journal of Productivity and Performance Management, 57(3), 223-236.
Wiese, D. S., & Buckley, M. R. (1998). The evolution of the performance appraisal process. Journal of Management History, 4(3), 233-249.
Wilson, J. P., & Western, S. (2000). Performance appraisal: An obstacle to training and development? Journal of European Industrial Training, 24(7), 384-390.
Youngcourt, S. S., Leiva, P. I., & Jones, R. G. (2007). Perceived purposes of performance appraisal: Correlates of individual- and position-focused purposes on attitudinal outcomes. Human Resource Development Quarterly, 18(3), 315-343.
Muhammad Zahid Iqbal
Department of Management Science,
COMSATS Institute of Information Technology,
Park Road, Chak Shahzad, Islamabad, Pakistan
E-mail: [email protected]
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer
Copyright Universiti Sains Malaysia Press Jan 2012