Co-constructing Performance Indicators in Home and Community Care: Assessing the Role of Nongovernmental Organizations in Three Canadian Provinces

Headnote

ABSTRACT

This article examines the extent to which public servants interact with non-governmental organizations (NGOs) to co-construct performance indicators in the home- and community-care sector. It uses 32 intensive qualitative interviews with NGO representatives and public servants in three Canadian provinces (British Columbia, Saskatchewan, and Ontario) with distinctive home- and community-care systems to uncover the experiences of NGO/government interactions around this issue and seeks to gain a greater understanding of the role of NGOs in shaping performance indicators. Varying funding and delivery models of home and community care across provinces put NGOs in different roles in the delivery of home and community supports, and hence, set different contexts for NGO/public servant interactions across the three provinces.

RÉSUMÉ

Cet article examine dans quelle mesure les fonctionnaires publics interagissent avec les organisations non gouvernementales (ONG) pour établir des indicateurs de performance dans les secteurs des soins à domicile et de proximité. Il se base sur 32 entretiens en profondeur avec des représentants d'ONG et des fonctionnaire publics dans trois provinces canadiennes (Colombie-Britannique, Saskatchewan et Ontario) ayant des systèmes distincts de soins à domicile et de proximité, et ce afin d'en apprendre davantage sur la réalité des interactions entre les ONG et le gouvernement. L'article cherche en outre à mieux comprendre le rôle des ONG dans la formulation des indicateurs de performance. Les divers modèles dans chaque province pour financer et offrir des soins à domicile et de proximité ont un impact sur la manière dont les ONG peuvent fournir leur aide à domicile et dans la communauté, et créent ainsi des contextes différents dans chacune des trois provinces pour les interactions entre les ONG et les fonctionnaires publics.

KEYWORDS / MOTS CLÉS: Community care; Nonprofit healthcare providers; Home care; Performance indicators; Policy engagement; Policy co-creation / Soins de proximité; Prestataires de soins à but non lucratif; Soins à domicile; Indicateurs de performance; Engagement politique; Co-création de politiques

INTRODUCTION

Canada is not alone in its interests in performance measures. Many countries, including the UK, Australia, and the USA, and international organizations, such as the World Health Organization (WHO) and the Organisation for Economic Co-operation and Development (OECD), have developed performance tools with the goals to enhance effectiveness, equity, efficiency, and quality as well as to meet the public's demand for increased transparency and accountability from their public institutions (Aboriginal Health & Medical Research Council of New South Wales, 2013; Arah, Klazinga, Delnoij, Ten Asbroek, & Custers, 2003; McLoughlin, Leatherman, Fletcher, & Owen, 2001; McPake & Mills, 2000; Pal, 2000). As Paul Thomas (2006) aptly expressed, performance measurements achieved popularity in large part to address serious "democratic deficits," (p. 6)-a decline in legitimacy and public confidence in political institutions along with other financial, social, and performance deficits.

This article examines the extent to which public servants interact with non-governmental organizations (NGOs) to co-construct performance indicators in the home and community care sector. Based on 32 extensive qualitative interviews with NGO representatives and public servants in three Canadian provinces (British Columbia, Saskatchewan, and Ontario) with distinctive home and community care systems, the study uncovers the experiences of NGO-government interactions. Specifically, it seeks to understand the role of NGOs in determining performance indicators by asking why performance indicators matter, what hand NGOs have in shaping them, under what contexts, and to what effect? Determining the appropriate indicators for the home and community care sector can play an important role in influencing the future direction of provincial healthcare systems, especially as the Canadian population ages. Varying funding and delivery models of home and community care across provinces put NGOs in different roles in the delivery of home and community supports, setting different contexts for NGO-public servant interactions across the three provinces. This article takes these different contexts into account in assessing how, and to what extent, NGOs influence the nature of performance indicators.

The literature offers a number of common definitions of performance indicators. According to Jan Mainz (2003) indicators are:

measures that assess a particular health care process or outcome; quantitative measures that can be used to monitor and evaluate the quality of important governance, management, clinical, and support functions that affect patient outcomes, and measurement tools, screens or flags that are used as guides to monitor, evaluate, and improve the quality of patient care, clinical support services, and organisational function that affect patient outcomes. (p. 524)

Les Pal (2014) offers a more succinct definition: performance indicators refer generally to "some measure of how well a service or activity is doing, either through financial or output measures or client satisfaction" (p. 174).

Researchers have long posed questions around the validity, reliability, impact, and scope of performance indicators in health. For example, there are ongoing debates regarding what can be measured, depending on how narrowly or broadly we define healthcare and our responsibility for it: how performance is conceptualized and measured (Arah et al., 2003); whether the emphasis is on "process" or "outcomes" (Mant, 2001; Steele Gray, Berta, Deber, & Lum, 2014a, 2014b); whether "quality" takes on a multidimensional nature, hence, requiring multiple measurement dimensions (Aboriginal Health & Medical Research Council of New South Wales, 2013); whether client preferences are part of the cost equation (Donabedian, 1988); how performance data are used; which stakeholder viewpoints should hold sway since different stakeholder have different views as to what processes or outcomes should be measured and how (Arah et al., 2003), while guarding against Karen Van Peursem, M.J. Pratt, and Stewart Lawrence's (1995) warning that "performance indicators are deceptive because they falsely convey an impression of objective truth" (p. 34) and in so doing, make certain aspects of performance visible while marginalizing other activities. Determining suitable performance indicators specifically for the home and community care sector is thus not just a technical matter, but foremost a political issue.

The subject of performance indicators is timely and topical particularly against the backdrop of an increased demand for evidence-based policymaking, lean practices and accountability, transparency, and better value for money within the context of public sector austerity. Along with a focus on quality improvement within healthcare systems and the need to assess the impact on the healthcare on the general population, are neoliberal principles of competition, laissez-faire, efficiency, productivity, profitability, and individual autonomy (Larner, 2000). The dominance of neoliberal paradigms in public policymaking has had profound implications for NGOs as neoliberalism positions non-profits as the prime agent for alternative service delivery (Shields, 2002). The interest of both government and nonprofit providers in social policy "fostered the development of an interdependent partnership" (Elson, 2007, p. 46). As a result, a relationship based on mutuality emerged as government needed services to be delivered and the funding of nonprofit agencies to do so provided decision-makers with a window on community needs and trends. Nonprofits, for their part, derived a source of financing and were theoretically in a position to influence policy (Brock, 2000). Nonprofit agencies, such as the Red Cross Home Care Services, the Victorian Order of Nurses, Saint Elizabeth's Visiting Nurses, and VHA Home Healthcare, to name but some of the larger entities, have deep historical roots in providing various community-based services. Being structured, however, by a neoliberal governance framework created a system of dynamic market competition between delivery agencies characterized by "time-limited contracts, legal control and accountability" (Eikas & Selle, 2002, p. 48).

As Bryan Evans, Ted Richmond, and John Shields have discussed elsewhere, neoliberal ideas and New Public Management (NPM) as its public administration counterpart have transformed how nonprofit agencies in general produce and deliver their core services (Evans & Shields, 2002; Evans & Shields, 2010; Evans, Richmond, & Shields, 2005; Richmond & Shields, 2004). NPM is particularly focused upon measuring performance (directed toward quantitative measures) with the promotion of discipline and parsimony in the allocation of resources (McLaughlin, Osborne, & Ferlie, 2002). "Doing more with less" has become the standard mantra for the operation of nonprofit service providers (Baines, Campy, Cunningham, & Shields, 2014).

The concept of performance indicators is especially controversial and contentious for NGOs in the home and community care sector. Increasing numbers of people including children with multiple chronic or complex care needs, rising health-service use across all age groups, more expensive medical technology, and demographic shifts toward an aging population, have prompted growing concerns about escalating healthcare costs and the sustainability of a publicly funded system in all Canadian provinces. As much as Canadians boast about healthcare as a "sacred trust," ensuring the highest quality of safe care within fixed budgets and the "value" of interventions for the money spent dominate debates about health spending. NGOs in the home and community care sector support clients to stay at home with such non-medical services as personal care (bathing, dressing, and feeding), homemaking, meal preparation, medication reminders, friendly visiting, transportation, security checks, recreation/social programs, and respite services, as well as day programs (Hollander & Chappell, 2002; Teplitsky, Williams, Deber, Lum, & Salib, 2006). The challenge for NGOs responsible for such mundane, everyday activities is to demonstrate the value of such supports to the healthcare system through performance indicators (Steele Gray et al., 2014a, 2014b).

WHY PERFORMANCE INDICATORS IN HOME AND COMMUNITY CARE?

Government spending in healthcare accounts for the largest share of provincial budgets. Performance indicators are consequently increasingly important to demonstrate "worthiness" for funding. If evidence and research guide political decisions about what to fund and what not to fund, how indicators and measurements get decided, and what performance gets measured are critical factors affecting the shape and direction of the healthcare sector. In 2010, the Canadian Institute for Health Information (CIHI) launched a national quality-improvement initiative called the Canadian Hospital Reporting Project (CHRP), the main purpose of which was to establish health performance indicators beginning in the acute care sector, so as to provide comparative facility-level information across hospitals to identify areas for better and more cost-effective care. After extensive research and consultation the project established 21 clinical performance indicators to measure effectiveness (quality and outcomes), patient safety, appropriateness, and accessibility; and six financial indicators to gauge efficiency and productivity (CIHI, 2011).

The acute care sector, which receives the largest proportion of provincial healthcare budgets, has determined that evidence-based, clinical metrics and benchmarks make sense. Measures of quality are closely aligned with accounting for how public funds are spent in that the impact of care is measured by quantifiable indicators such as wait times, numbers of beds, number of surgeries, number of doctors/nurses per population. However, what performance indicators make sense for home and community care? More importantly, who has input in deciding these indicators? This project examines the extent to which public servants interact with NGOs to "co-construct" (Evans & Sapeha, 2015) indicators that are applicable to home and community care, and the extent to which NGOs judge they have an impact in shaping these policy indicators.

METHODS

This study is part of a larger Social Sciences and Humanities Research Council (SSHRC) funded project, Policy Work in the Provinces: The "Production" of Policy Analysis and Advice in Canada's Provincial Public Services. Against the backdrop of the new governance literature that suggests "policies can no longer be struck in isolation in government" (Lindquist, 2009, p. 9; Osborne, 2010), this project examines how rank and file policy workers situated in three provincial governments (Ontario, British Columbia, and Saskatchewan) engage with their relevant non-governmental organization (NGO) counterparts in the policymaking process (Evans & Wellstead, 2013). While there is considerable research on the role and influence of public servants in shaping policy, there is considerably less written on the policy role and influence of NGOs as non-state actors. We focused on the policy interaction between public servants as state actors and the broad non-governmental organization (NGO) sector in shaping the policy process for determining performance indicators in a specific health sector: home and community care.

We defined NGOs broadly. NGOs in home and community care may include nonprofit community service agencies that deliver a range of non-medical home and community care supportive services to vulnerable populations, primarily older people who need assistance with the activities of daily living to live as independently as possible in the community. Other needs groups include persons with disabilities and growing numbers of medically fragile children and their families. The range of services provided varies among organizations. NGOs may also include non-provider organizations, such as unions, umbrella associations representing provider agencies, as well as voluntary citizens' groups. These NGOs may monitor policies in home and community care, raising their members' concerns to government and have advocacy and education as their main mandates.

For the NGO sector, we used web searches as well as a snowball sampling technique to identify different types of NGOs (e.g., provider NGOs such as community service agencies, non-provider advocacy organizations, unions, and associations) in diverse geographical locations, serving predominantly urban or rural areas. In our sampling technique, we also took into consideration the breadth of services provided (seniors only/across age categories; single or multiservice agencies) across the three provinces.

Public servants in sectors responsible for service planning, funding, and overall policy around home and community care were identified through web searches, as well as a snowball sampling technique through NGOs, academic, and professional contacts.

We used qualitative semi-structured phone interviews of approximately 90 minutes in length. The interviews were conducted between February 2012 and July 2013 using separate interview schedules with open-ended questions for NGO and government officials. The interviews were recorded, transcribed, and thematically coded using QSR NVivo 10. A total of 32 interviews were conducted: 16 with NGO representatives and 16 with public servants. All 32 interviews were used in this study. Table 1 presents the provincial breakdowns.

As a qualitative study with a limited sample size, we do not claim to present representative findings, but rather to identify relevant themes drawn from the experiences of NGOs and government officials that can help inform our understanding of the way quality indicators are co-constructed in home and community care, and in particular the role of NGO actors in this process.

SAMPLE CHARACTERISTICS

In Ontario, NGO respondents are relatively large (as measured by budgets over $5 million), multiservice agencies that provide a fairly comprehensive basket of front-line services to seniors, and rely primarily on government funds to sustain their operations. Ontario provider NGOs are also assisted by the Ontario Community Support Association, a large province-wide association that represents the common goals of members and advocates on their behalf to government.

The NGOs interviewed in BC are generally small (budgets under $2 million) to medium-sized (budgets from $2 million to $5 million) organizations targeting supports for multi-generations, and not dedicated solely to seniors. These NGOs draw funding from multiple sources including regional health authorities, different provincial ministries, local municipalities, charitable foundations, BC Community Gaming Grants, and donations. In comparison to Ontario, the supports to older people cover a much narrower range of services and may include social programs, volunteer shopping, transportation, meals-on-wheels, and friendly visiting. There is an association in BC (BC Care Providers Association) representing social services providers; however, in contrast to Ontario, respondents in BC either were not aware of the association, or did not consider it to be an effective voice in influencing policy.

In Saskatchewan, with a few exceptions, home and community care providers are mostly government employees. NGOs interviewed in the study are mainly non-providers and tend to be small organizations (budgets under $2 million) focusing on broad policy issues, such as advocating for a province-wide senior's strategy, pension issues, or supporting local health clinics and social programs. NGOs rely on multiple sources of funding, such as municipalities, regional health authorities, and donations. Some NGOs receive small amounts of funding from regional health authorities for community outreach, health promotion, and, in exceptional cases, to provide direct care services. There is no province-wide NGO representing community care; however, unions representing providers draw support from their national organization. Table 2 presents the characteristics of the NGO sample for all three provinces.

The respondents within government were policy analysts, planner, managers, and directors in the Ontario Ministry of Health and Long-Term Care, the BC Regional Health Authorities, and the Saskatchewan Health Authorities.

HOME AND COMMUNITY CARE MODELS IN THREE PROVINCES

Key to understanding the nexus between non-elected policy actors within government and NGO actors is to examine the models under which home and community care is funded and delivered within each of the provinces. Under the Canada Health Act (1984), only "medically necessary" hospital and doctor services are publicly funded. Home and community care services are not public entitlements under the Act. As a result, the availability, eligibility, access, costs, delivery, and range of home and community care services vary widely both within and across provinces.

Saskatchewan has a public provider model for home and community care. The Saskatchewan Ministry of Health provides global funding to the province's 12 regional health authorities (RHAs), which in turn, are responsible for funding and providing home and community care. Each RHA manages and employs providers to deliver professional services, such as nursing care and home support services including personal care, housekeeping, meal preparation, shopping, and respite care directly to clients. The regional health authorities do not generally contract out services to private non-profit or private for-profit agencies to deliver home and community care, although private providers exist and people may choose to buy private care if they wish (Canadian Health Care Association, 2009). In exceptional cases, the RHA will fund an NGO to deliver support services where there is an identified gap in community support services not provided by government, but on the whole, there are few provider NGOs in this province. Other NGOs that advocate for home and community care do so under the rubric of a variety of issues relating to the quality of life of seniors.

In contrast, British Columbia uses a mixed model. The Ministry of Health devolved responsibility for managing, monitoring, and funding the delivery of home support services to five regional health authorities. These regional health authorities sometimes directly provide home support services, and, at other times, contract out non-professional support services to private for-profit or not-for-profit agencies to provide a range of home support services, which include assistance with getting up and around, getting dressed, using the bathroom, preparing meals, and taking medication. In the mid-1990s, BC tightened its eligibility criteria so that clients requiring minimal levels of support now receive few or no publicly funded services. Except for individuals with the highest level of assessed needs, supportive services, such as housekeeping, grocery shopping, and transportation, are generally not available through the provincial home support program (BC Office of the Ombudsperson, 2012). Individuals who are not assessed at the highest level of care needs are thus left to buy supports from private providers.

In 2012, the BC Ministry of Health initiated a $15 million, three-year Better at Home province-wide program as part of Improving Care for B.C. Seniors: An Action Plan. The funds, managed by the United Way of the Lower Mainland, are designed to support simple, non-medical, day-to-day tasks, such as housekeeping, grocery shopping, home repair, friendly visiting, snow shovelling, yard work, minor home repairs, and transportation to appointments (Lum, 2013). Such services are delivered through local, nonprofit NGOs. While Better at Home does not reverse the BC government's eligibility policy, it does recognize the importance of NGOs delivering non-medical community supports to help people stay at home.

Ontario differs from both BC and Saskatchewan. Under the Ontario model, the Ministry of Health and Long-Term Care funds 14 regional health authorities known as Local Health Integration Networks (LHINs), which in turn flow funds to Community Care Access Centres (CCACs) to provide single-entry coordination based on assessed needs (Ontario Ministry of Health and Long-Term Care, 2014). The CCACs contract out professional and home support services to private for-profit and private not-for-profit providers (Canadian Health Care Association, 2009). The LHINs also fund nonprofit NGO community service agencies directly to provide a range of services to the community clients who may not qualify for services from the CCACs. For example, the CCACs provide professional home healthcare services (e.g., nursing) free-of-charge to eligible individuals when budgets permit and services are available. Not-for-profit community NGOs provide mostly non-medical social support services, but because of limited budgets, they may charge user fees on a sliding scale geared to income, and may subsidize services for individuals with low incomes. Individuals who can afford it can purchase services privately from commercial for-profit providers. The types of services offered depend on the resource capabilities of individual NGO community providers. It should be noted that in December 2015, the Ontario Minister of Health released a discussion paper that proposes important structural changes to the health system. In particular, the document proposes eliminating CCACs while expanding the role of the LHINs so as to improve access to, and links between, primary care and other services, including home and community care (Ontario Ministry of Health and Long-Term Care, 2015). In June 2016, Dr. Eric Hoskins, Ontario's Minister of Health and Long-Term Care, introduced Bill 210, Patients First Act 2016, which provides the legislative framework to such effect. Bill 210 is in its First Reading as of June 3, 2016 (Legislative Assembly of Ontario, 2016).

THEMES

A number of specific themes and subthemes emerged from our analysis of the qualitative interviews. They fall under the two broad categories: 1) assessing the value of quality of care, and 2) collaborative governance versus top-down decision-making.

Assessing the value of quality of care

What assessments are deemed appropriate to measure quality of care for the home and community care sector? Our interviews uncovered universal agreement among both government and community care actors that "quality of care" is important. But public servants and NGOs place significantly different emphasis on its priority. Public servants think that while quality of care is "hard to argue with," it is also difficult to measure and there is a need to use quantitative indicators to demonstrate that public money is well spent. This sentiment is clearly articulated in the following statements by government officials:

The one thing I think that the community is more concerned about than maybe the acute care sector [is] ... quality of life measures. A lot of the measures we have ... are clinical (like pressure sores and stuff) ... With surgery: did you get surgery in a reasonable amount of time; what is appropriate; was it safe? And with community, especially with something like homecare ... you're talking about quality of life. I think that it's much harder to measure ... that's a tough one. My sense is governments typically don't feel a strong impetus to fund that kind of activity. There's a sense that it may be nice but not necessary. (Saskatchewan [SK] Government 1)

Well it's about public accountability ... And it's also about measuring performance. If you know that you delivered 8 million hours of home support to 125,000 clients, so what? Do you know if that made any difference? Is that a good use of public dollars? Did you achieve any outcomes that you wanted to achieve for clients that cost hundreds and hundreds of millions of dollars? (British Columbia [BC] Government 1)

In contrast, home and community care NGO respondents place a premium on quality indicators. NGOs, particularly NGO service providers, believe that reporting on "difficult to quantify" outcomes should be an integral component of "accountability" measures, and express frustration that current performance indicators tend to focus primarily on "process" measures that tally "inputs" in quantifiable units: how much was spent, how many people were served, and for how many minutes.

What CCACs report on right now is evidence around administration . how money is spent, how many seniors are served. ... They are not measuring indicators that have to do with outcomes ... it's all process driven. (Ontario [ON] NGO 1)

My sense right now is that the government of Saskatchewan might pay some lip service to quality and community care but have done very little to actually measure it. I know that this long-term care report came out ... there's nothing to measure ongoing care. (SK NGO 3)

NGOs argue that performance indicators should also encompass "quality of life" outcomes such as the degree to which providers can promote choice, autonomy, dignity, comfort, security, peace of mind, empowerment, relationships, social activity, and enable a person to live independently at their highest physical, mental, emotional, and social potential (Organisation for Economic Co-operation and Development and European Commission, 2013). In fact, many respondents distinguish between "quality of care," which highlights the provider, and "quality of life," which shifts the focus to clients. NGO respondents report that current accountability requirements either overlook or relegate quality of life outcomes to an inferior status versus quantitative process measures. NGO respondents mostly agree that quality of life indicators are not as easy to identify and define as quantitative measures, such as a successful hip replacement. Nonetheless, in the interest of tracking and improving their own performance and the impact on clients, a number of NGOs have established internal quality indicators primarily through qualitative client and caregiver feedback surveys.

... It's to keep the client at home comfortably and safely so a lot of our indicators ask ... how we are able to keep clients independent safely, with security and peace of mind. As well, there is life satisfaction ... level of independence ... do you feel you can manage better at home with help? There is community engagement and social activities-how often do you feel that you can participate in the community? A lot of our support services, like transportation are designed to help people to be social and to participate in the community activities as much as they want, all of which contributes to their quality of life. (ON NGO 3)

... [W]e do Meals on Wheels ... aside from the quality of the meals: Are they happy with the meals? Are they happy with the diversity of food that they get? So we have some discussion around the food from the clients' perspective. Related to meals is isolation. A lot of our clients deal with issues of isolation ... often the Meals on Wheels person is the only visitor that isolated seniors will see in a week. So we do take a baseline when clients start with us in terms of talking to them about their sense of social connectedness and sense of life satisfaction. We do review that with them on an ongoing basis; give them a call; we've sent out surveys. It's sort of self-diagnostic ... they provide a self-evaluation of how they're feeling in terms of social interactions and their overall sense of satisfaction. (BC NGO 3)

We're launching a new program with volunteers who deliver six to ten meals to clients, and spend some time with the client, do a quick health check to see if any physical or mental health issues are starting to emerge, because we try to have the same drivers delivering; looking at cleanliness of the home to see if we have any issues with hoarding; looking for dementia or any other signs of health deterioration. So, we're monitoring and increasing the level of social connectedness. (BC NGO 2)

Such activities are very grassroots, unsystematic, unstandardized, and range widely depending on the organization, region and province. By the same token, the qualitative surveys on those diverse activities are unique to the specific NGOs, making comparability across the sector difficult. For this reason, another respondent offered the following cautionary note.

... We are about supporting independence, quality of life ... which isn't as easy to define as when you're dealing with surgery. I would say we lack of research in our sector ... our indicators are not standardized. So we can't say what our work means to the other parts of the system. We've got major, major catch up to do!

... Even if we defined quality indicators appropriate to our sector, we would be challenged to report on them. The community sector does not have the resources or expertise ... we don't have clinical practice leaders that are funded, like the hospitals. The resources to actually do the work should be underscored ... because it is a huge challenge. (ON NGO 3)

Size matters

While all NGOs in our sample recognized the value of quality indicators, there was a difference in outlook between larger versus smaller NGO providers. Smaller NGOs with resource limitations expressed concern that a proliferation of indicators can undermine their capacity to deliver their core services, in turn risking their ability to fulfill the quantitative requirements (e.g., number of visits and units delivered) that are currently central to their funding contracts with government. For example, some respondents noted that tracking quality requires information technology (IT) support, which is not covered in their contracts. Failing to meet the prescribed accountability measures can result in cancelled contracts; not meeting quality indicators carries no comparable consequence. As one of the small NGOs noted:

... If we can't figure out some way of demonstrating our impact, we're nowhere ... we're going to be lost ... we've been using the interRAI Community Health Assessment (RAI CHA) data assessment tool for almost four years now and we've actually got some cumulative data. So, we were really trying to figure out how we can use that to our advantage . but there are too many organizations that are too small and aren't going to be able to do this kind of thing. (ON NGO 5)

Large provider NGOs or providers who are government employees, such as those in Saskatchewan, indicate that they have somewhat more capacity to devote to quality indicators. To optimize their relative influence on government, NGOs often collaborate with organizations across multiple sectors. For example, NGOs that work closely with the primary care sector (e.g., Community Health Centres, Family Health Teams) express that they have a better chance of "getting a heads-up on policies coming down the pipe" and "have an easier time circling up to Deputy Ministers, Assistant Deputy Ministers" (ON NGO 7) than if they did not have such connections. In fact, across Ontario, there is a trend toward mergers and consolidation among community support service NGOs not only to increase organizational efficiency by creating economies of scale to expand service offerings, but also to strengthen their ability to attract funding and to advocate for the sector (Babcock, Charles Chiu, Hofmann, Macrae, & Tremblay, 2012). Ontario NGOs also make use of their province-wide association, the Ontario Community Support Association (OSCA), which has close relationships with deputy ministers, assistant deputy ministers, and elected officials and is often invited to sit at consultation tables to provide input on community care matters. In British Columbia, NGOs have banded together from time to time to influence policies on specific issues within a regional health authority, although not as yet on quality indicators. In Saskatchewan, NGOs look to their national counterparts where they exist.

Funding matters

For some NGOs in BC and Saskatchewan, going beyond contractual requirements to report on quality of life outcomes is a deliberate strategy to demonstrate the value of their interventions to the clients served. Such a tactic is especially useful in attracting support from a greater diversity of funding sources, such as municipalities, foundations, and charities.

We now have Performance Quality Indicator initiatives for all of our funding regardless of the source. ... For us it's all about leverage. ... Needs are growing and government dollars are not. We need to continually position ourselves as a well-run charitable organization that can attract other dollars - donations, United Way, local ministry or federal ministry, corporate partnerships, fee for service, etc. (BC NGO 2)

We can't count on government funding; we can't count on health region funding ... we need to get into measurements much more. We need to sell our story better ... we're working on key performance indicators. The United Way is funding community agencies, and we recognize the need for this, to be better at outcome measurement of specific programs so that we can sell our story better. (SK NGO 2)

As funding from government sources shrinks, NGOs in BC and Saskatchewan feel less willing to compromise when it comes to outcome factors for quantitative indicators. Instead, they prefer to increase the profile of their organization's attention to quality in the delivery of supportive services as key to leveraging funding from alternate sources. Ontario NGOs, by contrast, display little inclination to deviate from measures that are critical to accountability frameworks.

In sum, NGOs believe that appropriate outcome indicators are essential to the services they provide, but are difficult to construct in large part because of the inherent nature of preventative and supportive care. Public servants acknowledge the importance of quality/outcome measures, but in the absence of concrete and reliable indicators, fall back on politically salient process and outcome measures that are easy to gather, interpret, and communicate. Concepts such as "quality of life," independent living, and optimizing a person's physical, mental, emotional, and social potentials are too loose to fit easily within current government accountability frameworks. As one government policy worker confided, governments want demonstrations of effectiveness and not "one-off stories." The respondent went on to say that community supports may be "nice but not necessary" when governments are trying to reduce costs and increase efficiency, and added the comment that older people without supportive services may be institutionalized quicker is "a bit of a stretch" and not easily proven. (SK Government 1)

Collaborative governance versus top-down decision-making

Key questions addressed in this section are: What input do NGOs feel they have in shaping the measurements relevant to their sector? Is the process seen to be "predetermined"? A common refrain from respondents, regardless of province or government/NGO affiliation, is the recognition that ultimately, the government ministry has the final word in determining policy. While consultation is seen as important to the policy process by both sides, public servants and NGOs have contrasting views on the purpose and potential outcome of such consultations (Evans & Shields, 2014). Public servants are particularly interested in using consultations to encourage "buy-in" around decisions involving multiple stakeholders and believe that this kind of stakeholder engagement is the sine qua non of "collaborative governance" (Ansell & Gash, 2008). The process is seen to be critical to increase transparency and accountability, and is deemed helpful to reduce downstream implementation failures.

We do consult ... the policy process [is] not "predetermined". it's more of a balancing between the "wants" of different stakeholders and the government's mandate. If you ask me, the community sector is divided and is ineffective at communicating what the indicators for the sector should be. (ON Government 4)

We do a lot of collaborative work for their input. "What does this look like? Does this make sense to you?"-that kind of thing. At the end of the day the Ministry makes the decision but they have a great deal of input and we value their expertise. (BC Government 1)

NGO respondents, by contrast, are more skeptical about the value and impact of their consultative input, and admit that their ability to influence policy is severely limited in most cases. In fact, as an NGO representative expressed: "the indicators are somewhat imposed ... whatever [funding agreement] we sign ... is more focused on the contractual obligations versus the outcome of the clients" (ON NGO 7). Other NGO respondents reflected on their lack of influence in the consultation process as follows:

We were not at the table as participatory decision makers, hammering out what would be the appropriate plan. They talk to us but it's usually after the fact and they come out and say this is what they're doing now. (ON NGO 6)

It's very much unilateral ... whether I go to the affiliate's administration meeting or not. I do not see a lot of impact in what I bring to that meeting and things that I try to forward at that meeting in terms of influencing the health authority. (BC NGO 6)

Nonetheless, NGOs recognize that being "at the table" is still preferable to not being present. This sentiment was expressed in the following words:

Most agencies try to grab hold of the opportunity to go and say what we want to say ... I think for them [government representatives] it's a process of community engagement making them look nice, that we have consulted the community already . but whether they genuinely accept our ideas ... in any case, we try to participate. (ON NGO 3)

Do we participate fully, knowing what could be the outcome, or do we just stay outside of the tent and resist being part of things? ... [T]his is something that has always been a real issue for us in health care ... you need to have people present; you need to engage in this. It looks bad and it's unhelpful to not participate at all. (SK NGO 2)

Even though NGOs feel that their ability to play an influential role in shaping performance indicators is fairly minimal, they still believe in the importance of taking part in consultations with the regional health authorities and ministry representatives. Attending meetings is believed to be important to ascertain what policymakers are thinking and to be aware of the conversations and future initiatives taking place in their catchment areas. Most importantly, attending such meetings helps build good relationships with policy officials who are better positioned to formally influence government policy decisions.

THE POLITICS OF LEAN

A running theme from NGO respondents is that the financial climate of austerity in government presents additional obstacles for developing more appropriate evaluation tools for the community care sector. Quantitative measures make for politically appealing accountability sound bites in tight fiscal environments. For example, Ontario NGOs bemoan the government's current mantra that what can be measured gets funded, (Government of Ontario, 2005) since community care's core activities are not as easily quantifiable as the results for doctors and hospitals in the acute care sector.

As the "poor cousin" within healthcare, community care NGOs note that they lack the administrative and research capacity to develop appropriate and compelling qualitative measures. Devising surveys and analyzing their results takes time and diverts already stretched resources away from core service activities. Moreover, measuring "quality of life" is far more challenging than other types of "hard quantitative" indicators. Qualitative measures inevitably involve more subjective variables, which are both difficult to construct and more easily challenged. For these reasons, NGO respondents feel that they are unable to make a persuasive case for performance measures that adequately reflects the value of their services both to clients and to the broader health system.

Our downfall as a sector has always been that we haven't been able to make a strong case with real data [about] why we need to make community care a real focus. When the government and the ministry said, "what is measured, gets funded"... it's absolutely true. Without good indicators, it is really difficult for us to challenge the status quo ... it is easier for government and policy makers to be dismissive of what is still perceived as social support when you can't prove that Mrs. Smith didn't go to ER because she got ... in home services, home maintenance and meals on wheels. So much of community support is preventative care. (ON NGO 1)

As noted at the beginning of this article, neoliberal public management approaches have embraced "lean" production (Baines & McBride, 2015; Shields & Evans, 1998) focused on creating more value for clients and patients with fewer resources, and favours quantifiable performance measurements. While our public sector respondents fall back on justifying spending through easily measurable impacts, NGOs believe that their limited policy influence in defending the importance of supportive services has more to do with the "politics of lean" than with the lack of positive community care outcomes for clients.

The Ministry of Health has decided that we're all "going lean" so that's been a huge driver of improvement in the system right now. It's just being rolled out ... they're going to be soliciting their [Homecare employees] input through that process ... they would say that ... it's about improving the patient experience. That's how all of this is being sold. That's the messaging the government has. Putting the patient first and taking all of the waste and delays out of the system so that the system can focus solely on the patient. I'm skeptical. I'm pretty skeptical about what's going on here. (SK NGO 2)

The health authorities are constantly trying to cut us back and claw back money from us, so we have to be on our guard with these folks. Quite frankly, they are not partners of ours anymore ... but what happens is they're the messengers of the bad news. They have to come and say, "Well, we're really sorry, but they cut our budget back this year. Can you provide the same level of service?" (BC NGO 3)

Austerity not only pushes governments toward quantifiable outcomes, but according to NGOs, encourages governments to use accountability measures as a way to justify further cuts to community-based supportive services.

STRUGGLING TO BE HEARD

NGOs across the three provinces were clear that their policy voices are often not very effective. Yet, they persist in pulling and tugging at the policy process by targeting specific policy actors, the media, and the general public. The study asked the following question to NGOs: "How do you seek to influence home/community care policy generally, and quality indicators specifically? Do you target politicians, the minister, or the minister's staff, other senior public servants? What other strategies do you use? Not surprisingly, the responses varied across provinces.

Ontario NGOs seem to direct their efforts toward elected officials, their political staff, and high-ranking deputy ministers, assistant deputy ministers, or directors. They favour telling good news "performance stories" (Thomas, 2006) to the elected officials in their ridings so as to put the activities of the community sector on the public radar. Such stories help shine a positive light on government initiatives, showing that public money, when invested in the community care sector, is being put to good use.

I don't want to say they're [policy analysts] out of the picture ... but ... the people we are dialoguing with are at the ADM [assistant deputy minister] level, the director level ... we don't have much interaction with the people at management level or below. Part of the reason is that the status quo is easier than change, and there is not a lot of energy or enthusiasm at a bureaucratic level to think outside the box. So I often feel like you're banging your head against the wall and I don't waste time . I mean there are meetings you get called to and there is information that we share but I don't spend a lot of time to try and develop policy at the bureaucratic level ... (ON NGO 1)

Given the importance of building trust and sustaining open channels of communication with political staff, Ontario NGOs find it extremely frustrating when ministers, deputy ministers, assistant deputy ministers, and other public sector senior managers staff change. From a public administration perspective, "horizontality" or the lateral movement of public servants across the public sector is an appealing concept; horizontal mobility is seen to encourage innovative, "out-of-the-box" solutions and un-siloed ways to think about wicked problems (Bakvis & Juillet, 2004). Horizontal moves are not as positive for the community care sector; however, as they trigger the need for a new round of "education" as the community care file passes to a new set of policy people.

Fostering positive government relations is a huge part of the work that I do . it requires ongoing dialogue and communication with the government. ... When you represent the home and community sector . there is a lot lacking in the government's understanding and valuing of what that means for the community that we live in. So it's not just being able to advocate for good policy, but also having a huge role in education. ... Just as you build a relationship with someone, say, a policy adviser, and they are starting to understand the idiosyncrasies of the community care sector, they're gone, and then you have a new person and you're starting all over again. (ON NGO 1)

BC NGOs view elected officials and provincial policy workers as being remote. Consequently, BC NGOs tend to focus on public servants in the regional health authorities, or local level representatives with whom they have longstanding relationships and share values that have evolved over time.

I would say that we have very little opportunity to influence government policy. We very much are disempowered and we don't have the ability to negotiate contracts like we once did. We're very much dealing with representatives at the local level, but those local representatives from governments are not the policymakers. We work really hard and work collaboratively with the local level. So that may be Public Health, Home Health representatives, representatives from the hospital, representatives from the funders, the Health authorities. Those are the local people working really well together. Everybody's on the same page philosophically; shared vision, shared mission. But in terms of influencing policy; in terms of the direction of funds, the dispersal of funds, the priority of dollars, even saving and the efficiency of ideas in brainstorming, those opportunities haven't been around for a long time for not-for profits. (BC NGO 2)

Increasingly, BC NGOs sense that regional and central authorities tend to play one off against the other. Instead of identifying key public sector policy people to work with, they feel caught in a series of firewalls between regional and central authorities with one level diverting responsibility to the other level to justify decisions within diminishing resources.

In Saskatchewan, where NGOs are mostly non-providers, NGOs feel that their most effective strategy to shape the direction of the policy process is to rally citizens' groups, mobilize public opinion, especially around elections, and to harness media interest.

When we're defending a service or wanting a service improvement we want the public to know what we're advocating for. We want the public to support our agenda, like keeping homecare public or expanding long-term care. We want to educate the public on seniors' issues. In the last few years we've been doing more and more polling and focus group testing of our messages to make sure people understand what we're saying about public services. (SK NGO 1)

DELIVERY MODELS AND POLICY ENGAGEMENT

The varying funding and delivery models of home and community care put NGOs in different roles in the delivery of home and community supports, and hence, set different contexts for NGO/public servant interactions across the three provinces. Generally speaking, provider NGOs report having greater access to public servants and/or elected officials than do non-provider NGOs, with NGOs in Ontario that deliver all home and community supports reporting the closest contact with upper-level public servants and/or elected officials, followed by BC nGo providers. Saskatchewan NGOs are mostly non-providers and report having infrequent contact with public servants. They feel they have little input regarding the policy process generally, and performance indicators, specifically.

Furthermore, provider NGOs say they are consulted for their input on a range of policy issues, including performance indicators, more so than non-provider NGOs. It seems that provider NGOs have built up a wealth of on-the-ground expertise, and are critical to the delivery of services. Hence, they are often "in the loop" when governments are considering changes in administrative or evaluative policies, and are invited to the table regarding policies relating to home and community care. By the same token, provider NGOs whose funds come mostly from government have a vested interest to take every opportunity to "be at the table" to influence policies that may affect their operational viability, with Ontario NGOs reporting the most extensive consultative relationships and Saskatchewan NGOs (mostly non-providers), the least. Finally, Ontario also has the advantage of a large, well-funded umbrella organization that advocates for the community care sector and provides a forum for NGOs to interact with one another.

The funding relationship between Ontario NGOs and government is not without the irony typical of co-dependent relationships. Although Ontario NGOs would at face value appear to have the greatest policy process input and impact on quality indicators for the home and community care sector, their dependence on government funding paradoxically undermines their capacity to shape performance indicators to be appropriate for their sector. For this reason, despite the much-vaunted place at the policy table for formal meetings or informal discussions, and despite their political leverage as providers, they feel their influence is minimal. As discussed above, their place at the policy table is a small one.

The relatively recent regionalization process in Ontario adds to the policy influence challenges of Ontario NGOs. Like in other provinces, the concept of regional administrative units was designed to be more responsive to the diversity of local needs and to be sensitive to variations in demographics, and regional peculiarities. It appears that in Ontario, regionalization has added another layer of policy people who must be "educated" and persuaded. With each personnel change, this education must begin again, echoing what others have identified as a loss of continuity and institutional memory (Bourgault, 2003). Finally, our respondents seem to concur that the additional layer allows the regional level to offload financially difficult decisions to the more distant central level of government, making efforts around shaping appropriate indicators even more challenging. In the end, indicators are framed in the context of provincial priorities, namely accountability and quantifiable indicators with a focus on acute care, and may not reflect the viewpoints of community NGOs regarding using indicators to illustrate how the sector supports independence through high-quality care. Given the dependence of NGOs on LHIN [Local Health Integration Network]-based funding for core services in Ontario, community agencies face challenges in providing input into quality indicators if funders are not receptive toward those priorities.

BC NGOs are forging a different path. With a longer history of regionalization as well as austerity measures that have delisted many community support services since the mid-1990s, BC NGOs are diversifying their funding sources to tap resources from different provincial ministries, local municipalities, charitable foundations, such as the Vancouver Foundation, and donations. Thus, BC NGOs are increasingly using performance indicators to validate their organizations' value as they seek funding support from diverse sources. In fact, the more varied the sources of funding, especially from non-traditional sources, the more "quality" is important as a "brand." For this reason also, accreditation is valued as a key characteristic in attracting funding, as quality is seen to be an important component of the accreditation processes. BC NGOs may also use quality indicators and evidence of quality improvement to strengthen their claims for increased resource allocation within their RHA (regional health authority). For the most part, the drive for appropriate quality indicators originates outside government/RHA policy processes.

Because Saskatchewan home and community care providers are mostly government employees, regional health authorities develop internal quality indicators as part of their mandate to ensure quality and safety. The NGOs interviewed state they do not have much influence over the discussion around quality indicators in community care. They are often community-based interest or advocacy groups with an interest in advocating on a broad range of issues, including the quality of life of seniors under home and community care. Nonetheless, recognizing the importance of performance indicators within fiscally lean times, these non-provider NGOs attempt to exercise influence by aligning with government providers to prevent the erosion of community care services as has happened in other provinces.

In Saskatchewan, interestingly, unions have emerged as an unlikely advocate for quality performance indicators. By arguing for the importance of supports that keep older people at home, unions are in effect advocating for protecting the jobs of government home and community care providers. In a period of government fiscal austerity, unions may, however, find themselves in a difficult position as they also represent workers in the acute care sector.

DISCUSSION AND CONCLUSION

In conclusion, we return to our central research question: what performance indicators make sense for home and community care and who has input in deciding these indicators? Our findings suggest that state actors do recognize the calls from the home and community care NGOs to develop appropriate performance indicators for this sector, and to incorporate quality of life as an essential component of performance measurements. However, far from establishing new paradigms of power sharing (Ansell & Gash, 2008), the NGOs in this study do not see themselves as effective participants in new governance structures (Bevir, 2011; Noveck, 2011). As evidence of the minimal effectiveness of NGOs in the policy process, one needs only to point to the ambiguous ways in which "quality" components have so far been incorporated into policies relating to home and community care performance measurements. Policy is still very much determined from senior government sources based primarily on measurable process inputs and quantifiable outcomes. New governance approaches to policy decision-making have failed to materialize and neoliberal approaches continue to dominate.

The Saskatchewan Home Care Policy Manual (2015) governs all aspects of the delivery of home and community care and empowers regional health authorities to develop and implement an evaluation system for performance measurements that are to include quality improvement for their home care programs. There is, however, no clear indication of the type of data required and how it should be collected. Furthermore, regional health authorities have considerable leeway in interpreting the nature of these quality improvement programs.

In British Columbia, the Ministry of Health Home and Community Care Policy Manual (2012) states that health authorities, "are required to use performance data to measure and monitor improvements in quality of care and health outcomes for home and community care clients" (p. 2). However, the Ministry of Health delegates its monitoring and enforcement responsibilities to the health authorities without requiring them to report on how they carry these out. Hence, beyond requirements for data submission from service providers, the policy lacks province-wide, specific standards, such as methods for conducting inspections, investigating complaints, and tracking and responding to reportable incidents. According to the BC Ombudsperson report (2012), the requirement for performance standards and measures is not considered to be legally binding. Individual regional health authorities may implement quality indicators and measures for internal decision-making processes.

In Ontario, community support service provider NGOs are required by law under the Local Health System Integration Act, 2006, to enter into Multi-Sector Service Accountability Agreements (MSAAs) with the LHIN funder. Under the 2014-17 MSAA, client experience based on client satisfaction surveys will be the only indicator that looks at the quality of care provided by the organization. An organization's contract cannot be cancelled based on the results of these surveys in the same way that it can if quantitative requirements are not fulfilled (LHIN, 2014).

Medicare in Canada with its principles of universality, accessibility, comprehensiveness, portability, and public administration remains a defining characteristic of our Canadian identity and, in comparison to a market-based healthcare system, has produced superior health status indicators. However, it has created an institutional and policy legacy that places physicians at the heart of health decision-making at all levels and underscores the widespread public (mis)understanding that equates healthcare only with hospital and doctor care (Tuohy, Flood, & Stabile, 2004). Home and community care providers delivered mainly by NGOs remain "outsiders" within the Canadian healthcare system (Eversole, 2010; Van der Heijden & Ten Heuvelhof, 2012).

It is important, however, not to draw the conclusion that NGOs have no impact on quality, even if they have minimal impact on policies relating to quality of life indicators. NGOs from our interviews distinguish between "big wins" and "small wins." The "small wins" come when policy hits the implementation road, below the operational radar where, as one respondent noted, "there is always 'wiggle' room!" (ON NGO 4) Community care providers confide that they can do much to affect their clients' quality of life by making decisions about how they deliver services on the ground. Such implementation decisions consider the perspectives and past practices of NGOs below and beyond the radar of public servants. As noted above, larger NGOs may have more wiggle room than small ones. The big question is whether the "small wins" can open the policy space for greater policy gains.

What then are the main messages of our respondents in the home and community care sector to public policy planners regarding performance indicators? The first key message is the need to recognize the variability of care organizations: there is no single best approach to measuring performance across the care continuum from hospital to home and community. The "gold standard" of random clinical trials is neither possible nor applicable in all settings (Howlett, 2009; Thomas, 2007; Weiss, 1993). While scientifically based objectivity and feasibility performance measures can be used in the acute care sector where the primary outcome is to cure, alternate, more qualitative "indicators" may be more appropriate to assess the impact of the community care sector where the emphasis is on helping with everyday living (Campbell, Braspenning, Hutchinson, & Marshall, 2003; Donabedian, 1988; Goodwin, Dixon, Poole, & Raleigh, 2011). Avedis Donabedian (1988) and other authors (Goodwin et al., 2011; Wright, 2012) have highlighted the importance of "hard to measure," more qualitative, or subjective elements of care on the outcomes of care. Furthermore, the multi-dimensional nature of "quality" may require multiple measurement dimensions (Aboriginal Health & Medical Research Council of New South Wales, 2013).

The second key message is not to throw the performance baby out with the measurement bath water. To date, NGOs rely on internal quality indicators to track and improve their own performance and on "good news stories" to publicize the effectiveness of their healthcare supportive interventions. However, governments can play a strategic role in effectively partnering with NGOs to co-construct appropriate community care indicators within an overarching systematizing framework so that resulting qualitative outcomes are comparable across organizations rather than being idiosyncratic and particular to specific agencies.

The marginal influence that the home and community care NGO sector has on developing performance measures in health belies the mounting evidence from other jurisdictions that quality care from this sector is critical to containing escalating healthcare costs (Commission on the Reform of Ontario's Public Services, 2012; Donner, 2015; Sinha, 2013; Walker, 2011). Evidence indicates that transitioning people seamlessly from hospitals to home results in better outcomes for those requiring multiple services from multiple providers, and makes healthcare systems more cost-effective and sustainable, in contrast to fragmented or "siloed" systems (Johri, Béland, & Bergman, 2003; MacAdam, 2008). Importantly, a thriving home and community care sector can also prevent unnecessary emergency room visits and help relieve the imminent caregiver gap and caregiver burden as Canada's demography shifts to an aging population (McNeil & Hunter, 2014; Williams, Lum, Morton-Chang, Kuluski, Peckham, Warrick, & Ying, 2016). The ability of NGOs to co-construct appropriate performance indicators is critical to a sustainable healthcare system, especially in the context of neoliberal austerity.

ACKNOWLEDGEMENTS

The authors wish to gratefully acknowledge the research assistance of Alvin Ying, the valuable comments from anonymous reviewers, and research funding from the Social Sciences and Humanities Research Council (SSHRC).

References

REFERENCES

Aboriginal Health & Medical Research Council of New South Wales. (2013). A literature review about indicators and their uses. Sydney, AU: Aboriginal Health & Medical Research Council of New South Wales. URL: http://www.ahmrc.org.au/index.php?option=com_docman&task=doc_download&gid=200&Itemid=45 [April 26, 2016].

Ansell, C., & Gash, A. (2008). Collaborative governance in theory and practice. Journal of Public Administration Research, 18(4), 543-571.

Arah, O.A., Klazinga, N.S., Delnoij, D.M., Ten Asbroek, A.H.A., & Custers, T. (2003). Conceptual frameworks for health systems performance: A quest for effectiveness, quality, and improvement. International Journal for Quality in Health Care, 15(5), 377-398.

Babcock, A., Charles Chiu, A., Hofmann, T., Macrae, S., & Tremblay, S. (2012). From strategy to implementation: An integration toolkit for community-based health service providers. Toronto, ON: Woodgreen Community Services.

Baines, D., & McBride, S. (2015). Orchestrating austerity: Impacts and resistance. Halifax, NS: Fernwood.

Baines, B., Campey, J., Cunningham, I., & Shields, J. (2014). Not profiting from precarity: The work of nonprofit service delivery and the creation of precariousness. Just Labour: A Canadian Journal of Work and Society, 22(Autumn), 74-93.

Bakvis, H., & Juillet, L. (2004). The horizontal challenge: Line departments, central agencies, and leadership. Ottawa, ON: Canada School of Public Service.

Bevir, M. (2011). Governance as theory, practice, and dilemma. In M. Bevir (Ed.), The SAGE handbook of governance (pp. 1-16). London, UK: SAGE Publishing.

Bourgault, J. (2003). The contemporary role and challenges of deputy ministers in the Government of Canada. Ottawa, ON: Canadian Centre for Management Development.

British Columbia. Ministry of Health. (2012). Home and community care policy manual. Chapter 3: Performance management. Victoria, BC: Ministry of Health. URL http://www2.gov.bc.ca/gov /DownloadAsset?assetId =9B75FD6EBF374E32960422F15345EC1C [October 1, 2015].

British Columbia. Office of the Ombudsperson. (2012). The best of care: Getting it right for seniors in British Columbia (Part 2). Public report no. 47 to the Legislative Assembly of British Columbia. Victoria, BC: Queen's Printer.

Brock, K. (2000). Sustaining a relationship: Insights from Canada on linking the government and the third sector. Paper presented at the Fourth International Conference of the International Society for Third Sector research, July 5-8. Dublin, Ireland.

Campbell, S., Braspenning, J., Hutchinson, A., & Marshall, M. (2003). Improving the quality of health care: Research methods used in developing and applying quality indicators in primary care. British Medical Journal, 326(7393), 816-819.

Canadian Institute for Health Information (CIHI). (2011). Health indicators 2011: Definitions, data sources and rationale, June 2011. Ottawa, ON: CIHI. URL: http://www.cihi.ca/cihi-ext-portal/pdf/internet/definitions _062011_en [October 1, 2015].

Canadian Healthcare Association. (2009). Home care in Canada: From the margins to the mainstream. Ottawa, ON: Canadian Healthcare Association. URL: http://www.cha.ca/wp-content/uploads/ 2012/11/Home _Care_in_Canada_From_the_Margins_to_the_Mainstream_web.pdf [October 1, 2015].

Commission on the Reform of Ontario's Public Services. (2012). Public services for Ontarians: A path to sustainability and excellence. Toronto, ON: Queen's Printer for Ontario.

Donabedian, A. (1988). The quality of care: How can it be assessed? Journal of the American Medical Association, 260(12), 1743-1748.

Donner, G. (2015). Bringing care home: Report of the Expert Group on Home and Community Care. Toronto, ON: Expert Group on Home and Community Care. URL: http://health.gov.on.ca/en/public/ programs/ccac/docs/hcc_report.pdf [April 26, 2016].

Eikas, Magne, & Selle, Per. (2002). A contract culture even in Scandinavia. In Ugo Ascoli & Costanzo Ranci (Eds.), Dilemmas of the welfare mix: The new structures of welfare in an era of privatization (pp. 47-75). New York, NY: Klumer Academic/ Plenum Publishers.

Elson, P. (2007). A short history of voluntary sector-government relations in Canada. The Philanthropist, 21(1), 36-72.

Evans, B., & Sapeha, S. (2015). Are non-government policy actors being heard? Assessing new public governance in three Canadian provinces. Canadian Public Administration, 58(2), 249-270.

Evans, B., & Shields, J. (2014). Nonprofit engagement with provincial policy officials: The case of Canadian immigrant settlement services and NGO policy voice. Policy and Society, 33(2), 117-127.

Evans, B., & Shields. J. (2010). The third sector: Neoliberal restructuring, governance, and the re-making of state-civil society relationships. In C. Dunn (Ed.), The handbook of Canadian public administration (2nd ed.), (pp. 305-318). Don Mills, ON: Oxford University Press.

Evans, B., Richmond, T., & Shields, J. (2005). Structuring neoliberal governance: The nonprofit sector, emerging new modes of control and the marketisation of service delivery. Policy and Society, 24(1), 73-97.

Evans, B., & Shields, J. (2002). The third sector, neo-liberal restructuring and "partnerships": Governance and the re-making of state-civil society relationships. In Chris Dunn (Ed.), Oxford Handbook of Canadian Public Administration, 139-158. Toronto, ON: Oxford University Press.

Evans, B., & Wellstead, A. (2013). Policy dialogue and engagement between non-government organizations: A survey of processes and instruments of Canadian policy workers. Central European Journal of Public Policy, 7(1), 73-97.

Eversole, R. (2010). Remaking participation. Community Development Journal, 47(1), 29-41.

Goodwin, N., Dixon, A., Poole, T., & Raleigh, V. (2011). Improving the quality of care in general practice. London, UK: The King's Fund. URL: http://www.kingsfund.org.uk/sites/files/kf/improving-quality-of-care-general-practice-independent-inquiry-report-kings-fund-march-2011_0.pdf [April 26, 2016].

Government of Canada. (1985). Canada Health Act (R.S.C., 1985, c. C-6). URL: http://laws-lois.justice.gc .ca/eng/acts/C-6/FullText.html .

Government of Ontario. (2005). Performance Measurement: A Reference Guide. Toronto, ON: Queen's Printer for Ontario.

Government of Ontario. (2007). Program Evaluation Reference & Resource Guide For the Ontario Public Service. Toronto, ON: Queen's Printer for Ontario. URL: http://www.ontla.on.ca/library/repository/ mon/20000/274278.pdf .

Government of Saskatchewan (2015). Home Care Policy Manual. URL: http://www.publications.gov.sk.ca/ details.cfm?p=22841 .

Government of Ontario. (2016). Bill 210: An Act to Amend Various Acts in the Interest of Patient-Centred Care (Patients First Act). 1st Reading June 2, 2016, 41st Parliament, 1st session. URL: http://www.ontla.on .ca/web/bills/bills_detail.do?locale=en&Intranet=&BillID=4054 [June 3, 2016].

Hollander, M.J., & Chappell, N. (2002). Final report of the National Evaluation of the Cost-Effectiveness of Home Care. Synthesis report. Victoria, BC: National Evaluation of the Cost-Effectiveness of Home Care. URL: http://www.homecarestudy.com/reports/full-text/synthesis.pdf [April 26. 2016].

Howlett, M. (2009). Policy analytical capacity and evidence-based policy-making: Lessons from Canada. Canadian Public Administration, 52(2), 153-175.

Johri, M., Béland, F., & Bergman, H. (2003). International experiments in integrated care for the elderly: A synthesis of the evidence. International Journal of Geriatric Psychiatry, 18, 222-235.

Larner, W. (2000). Neo-liberalism: Policy, ideology, governmentality. Studies in Political Economy, 63(Autumn), 5-25.

Lindquist, E. (2009). There's more to policy than alignment. CPRN research report. Ottawa, ON: Canadian Policy Research Networks. URL: http://www.cprn.org/documents/51384_EN.pdf [October 1, 2015].

Local Health Integration Network (LHIN). (2014, November). Multi-sector service accountability agreement 2014-2017: Indicator technical specifications. Version 5. Toronto, ON: LHIN. URL: http://www .mississaugahaltonlhin.on.ca/~/media/sites/mh/Secondary%20Navigation/For%20HSPs/MSAA%20CAPS /2015-06%20MSAA%20Indicator%20Tech%20Specs%20FINAL.pdf?la=en [December 21, 2015].

Lum, J.M. (2013). Did you know? BC's Better at Home program. CRNCC E-News, 31(Spring), 2-3. URL: http://www.ryerson.ca/content/dam/crncc/enews/pdfs/2013/2013-spring-crncc-enews-vol31.pdf [October 1, 2015].

MacAdam, M. (2008). Frameworks of integrated care for the elderly: A systematic review. Ottawa, ON: Canadian Policy Research Networks.

Mainz, J. (2003). Defining and classifying clinical indicators for quality improvement. International Journal for Quality in Health Care, 15(6), 523-530.

Mant, J. (2001). Process versus outcome indicators in the assessment of quality of health care. International Journal for Quality in Health Care, 13(6), 475-480.

McLoughlin, V., Leatherman, S., Fletcher, M., & Owen, J.W. (2001). Improving performance using indicators: Recent experiences in the United States, the United Kingdom, and Australia. International Journal for Quality in Health Care, 13(6), 455-462.

McLaughlin, K., Osborne, S., & Ferlie, E. (2002). New public management: Current trends and future prospects. Oxon, UK & New York, NY: Routledge.

McNeil, C., & Hunter, J. (2014). The generation strain: Collective solutions to care in an ageing society. London, UK: Institute for Public Policy Research. URL: http://www.ippr.org/files/publications/pdf/generation-strain_Apr2014.pdf?noredirect=1 [April 26, 2016].

McPake, B., & Mills. A. (2000). What can we learn from international comparisons of health systems and health system reform? Bulletin of the World Health Organization, 78(6), 811-820.

Noveck, B. (2011). The single point of failure. In S. Van der Hof & M. Groothuis (Eds.), Innovating government (pp. 77-99). The Hague, NL: Asserr Press.

Ontario Ministry of Health and Long-Term Care. (2014). Ontario's local health integration networks. Toronto, ON: Ontario Ministry of Health and Long-Term Care. URL: http://www.lhins.on.ca/aboutlhin.aspx [October 1, 2015].

Ontario Ministry of Health and Long-Term Care. (2015). Patients first: A proposal to strengthen patient-centred health care in Ontario. Toronto, ON: Queen's Printer for Ontario. URL: http://www.health.gov. on.ca/en/news/bulletin/2015/docs/discussion_paper_20151217.pdf [December 21, 2015].

Organisation for Economic Co-operation and Development and European Commission. (2013, June). A good life in old age? Monitoring and improving quality in long-term care. Paris, FR & Brussels, BE: Organisation for Economic Co-operation and Development and European Commission. URL: http://www.oecd.org/els/health-systems/PolicyBrief-Good-Life-in-Old-Age.pdf [October 1, 2015].

Osborne, S.P. (Ed). (2010). The new public governance: Emerging perspectives on the theory and practice of public governance. London, UK: Routledge.

Pal, L.A. (2014). Beyond policy analysis: Public issue management in turbulent times (5th ed.). Toronto, ON: Nelson Education.

Pal, L. (2000). "There ought to be a law!" Instrument choice: An overview of the issues. Ottawa, ON: Justice Canada, Research and Statistics Division. URL: http://justice.gc.ca/eng/rp-pr/csj-sjc/jsp-sjp/rp02_10-dr02_10/rp02_10.pdf [April 26, 2016].

Richmond, T., & Shields. J. (2004). NGO restructuring: Constraints and consequences. Canadian Review of Social Policy, 53(Spring/Summer), 53-67.

Shields, J. (2002, Februay 21). Capturing civil society: The third sector in the shadow of the state. Ryerson University, Toronto, ON: The SRC Sarwan Shahota Lecture.

Shields, J., & B. Mitchell Evans. (1998). Shrinking the state: Globalization and public administrative "reform." Halifax, NS: Fernwood Publishing.

Sinha, S.K. (2012). Living longer, living well: Report submitted to the Minister of Health and Long-Term Care and the Minister Responsible for Seniors on recommendations to inform a seniors strategy for Ontario. Toronto, ON: Queen's Printer for Ontario.

Steele Gray, C.A., Berta, W., Deber, R., & Lum, J. (2014a). Home and community care sector accountability. Healthcare Policy, 10(S), Article 2. URL: http://digitalcommons.mcmaster.ca/hro-ors/vol2/iss1Z2 [April 26, 2016].

Steele Gray, C.A., Berta, W., Deber, R., & Lum, J. (2014b). Seeking accountability: Multi-Service Accountability Agreements (MSAAs) in Ontario's community support sector. Health Reform Observer, 2(1), Article 2. URL: http://digitalcommons.mcmaster.ca/hro-ors/vol2/iss 1 /2 [April 26, 2016].

Teplitsky, F., Williams, P., Deber, R., Lum, J., & Salib, D. (2006). In-focus fact sheet: Community support services. Toronto, ON: Canadian Research Network for Care in the Community. URL: http://www.cmcc .ca/knowledge/infocus/factsheets/InFocus-CommunitySupportServicesfinal.pdf [April 26, 2016].

Thomas, P.G. (2007, May). Why is performance-based accountability so popular in theory and difficult in practice? [Paper]. Taipei City, Taiwan: World Summit on Public Governance: Improving the Performance of the Public Sector.

Thomas, P.G. (2006). Performance measurement, reporting, obstacles and accountability: Recent trends and future directions. Canberra, AU: ANU E Press.

Tuohy, C.H., Flood, C.M., & Stabile, M. (2004). How does private finance affect public health care systems? Marshaling the evidence from OECD nations. Journal of Health Politics, Policy and Law, 29(3), 359-396.

Van der Heijden, J., & Ten Heuvelhof, E. (2012). The mechanics of virtue. Environmental Policy and Governance, 22(3), 177-188.

Van Peursem, K.A., Prat, M.J., & Lawrence, S.R. (1995). Health management performance. Accounting, Auditing & Accountability Journal, 8(5), 34-70.

Williams, A.P., Lum, J., Morton-Chang, F., Kuluski, K., Peckham, A., Warrick, N., & Ying, A. (2016). Integrating long-term care into a community-based continuum: Shifting from "Beds" to "Places." Montreal, QC: Institute for Research on Public Policy. URL: http://irpp.org/wp-content/uploads/2016/02/study-no59.pdf [April 26, 2016].

Walker, D. (2011). Caring for our aging population and addressing alternate level of care: Report submitted to the Minister of Health and Long-Term Care. Toronto, ON: Queen's Printer for Ontario.

Weiss, C.H. (1993). Where politics and evaluation research meet. American Journal of Evaluation, 14(1), 93-106.

Wright, M. (2012). Pay-for performance programs: Do they improve the quality of primary care? Australian Family Physician, 41(12), 989-991.