Using comparative performance measurement

Abstract

Comparative Performance Measurement, by Elaine Morley, Scott P. Bryant, and Harry P. Hatry, is reviewed.

Elaine Morley, Scott P. Bryant, and Harry P. Hatry, Comparative Performance Measurement (Urban Institute Press, 2001). 98 pp., $28.00 paper.

How well is a program or agency performing in comparison to similar agencies, the private sector, or industry standards? The emphasis of Comparative Performance Measurement (CPM) is on comparing agencies against similar organizations and using the information to improve agency performance.

Written so the reader can readily walk through the steps of comparative performance measurement, including its limitations, the book is a good primer. It provides useful tools for the public administrator who may be new to CPM and is faced with undertaking CPM or understanding comparisons developed by outside parties. Masters of public administration instructors and students should also find it to be complementary to more theoretical work relating to public management.

According to the authors, the book's primary audience is practitioners at all levels of government, as well as organizations that provide similar public services. However, the examples emphasize state and local programs with frequent reference to the International City/County Management Association's (ICMA's) CPM Consortium project, which reports comparative performance data from over 50 participating cities and counties.

The first two chapters set the context. The authors provide a compelling rationale for using CPM as a motivator for improved performance, as an accountability tool, and as a tool for determining best practices. This is conveyed in the context of current public management improvement efforts, including performance measurement, which they wisely advise should precede CPM. The reader is also cautioned about the limitations of CPM: "Perhaps the most significant of these is the fact that no two jurisdictions or organizations are completely comparable" (5).

To set a framework for the types of comparative efforts, the second chapter explains CPM efforts in terms of comparisons conducted from four perspectives. Some examples in the first two categories (consumer-oriented and advocacy group comparisons) are well known-for example, Governing magazine's "Grading the States: A Management Report Card." While those kinds of comparisons are often cited and useful, the comparisons that public administrators typically have more influence over-or engagement in-are ones in the other two categories (mandated reporting and self-initiated comparisons). In such reporting, greater emphasis is placed on outcome indicators as well as on interjuris-dictional coordination in using the results. An example of the self-initiated comparisons is the ICMA Local Government CPM Consortium. ICMA reports provide a variety of comparative information on police, fire, neighborhood, and other services.

Chapter 2 concludes with a section on the characteristics of and differences among CPM efforts. It addresses such factors as the source of information used for comparison and the method for reporting comparative resuits. As the authors point out in a later chapter, reporting data without any explanatory information does not give a complete picture of what is impacting performance.

Chapters 3 through 7 cover the primary stages of CPM, beginning with determining the scope of a CPM effort and concluding with reporting its results. Good advice and relevant examples are provided, often in exhibits. One exhibit in chapter 3 identifies performance measurement definitions. These definitions may seem obvious, but failure to adhere to them may result in confusion. For example, other exhibits in chapter 3 use the subtitle "illustrative outcome indicators," but several of the indicators do not appear to be outcome indicators-for instance, total number of new low- to moderate-income housing units completed or cost per centerline mile repaved by contractor. Also, the terms "performance indicator" and "performance measure" are frequently used interchangeably throughout the book.

Chapter 4 explains how to define performance indicators in measurable terms in three steps. This guidance should help readers new to CPM in developing more viable performance indicators. In introducing this chapter, the authors imply that having data roughly right is preferable to having no data. The potential consequences of using CPM information to make allocation and program policy decisions are significant. Further qualification could have included explaining the importance of confirming that the comparisons are valid within some acceptable range.

Collecting data for comparison is covered in chapter 5. It follows the instructional approach of the previous two chapters, including sharing cautionary advice. "Data collection deserves considerable attention in all CPM efforts" (60) is the concluding remark. That statement should not be taken lightly; it applies to data collected internally for self-initiated efforts, as well as for externally driven CPM efforts.

Determining how well an agency is doing is the subject of chapter 6. Readers are appropriately reminded that a major purpose of CPM is to provide information that managers and others can use to improve performance. A critical aspect of CPM is to assess why some agencies do better than others. The statistical tests identified should be seriously considered to determine if the differences in outcomes are significant and whether outside factors are influencing results. Understanding what contributed to the results and what difference the numbers make should precede the next step, which is reporting results.

Publicly reporting on performance promotes accountability and encourages agencies to make improvements. However, it can also raise questions that cannot readily be answered, increase the potential for misunderstanding and misreporting, and thwart performance-based endeavors. Chapter 7 covers how agencies can clearly report comparative information in documents, including using charts, tables, and other visual aids. To a lesser extent it advises on how to report bad news honestly.

In the last chapter, the authors discuss how agency managers can effectively use comparative performance information, once its accuracy is established. They discuss 10 uses, starting with instituting corrective action and concluding with communicating with the public. Although we hear a lot about government being more customer-oriented, the authors observe that relatively few communities use CPM data in their citizen involvement efforts. A final note at the end of this book explains that many nonprofit service organizations are also taking advantage of CPM.

Many readers will appreciate this straightforward guide to understanding and applying CPM to advance public-service management and decision making. Others may be looking for a more in-depth discussion on conducting CPM and find the guidance in this book limited.

Public administrators are often asked tough questions, such as: What is the cost in other jurisdictions to house jail inmates or reduce property crimes? What is the success rate of other states in reducing child abuse? What is the standard industry cost per successful client of drug treatment programs? Answers to these questions require an understanding of how the reported data was developed and how to make adjustments so the numbers are reasonably comparable. This book provides a solid foundation to build upon as we seek answers to these difficult questions.