Evaluating an organization-wide disparity reduction program: Understanding what works for whom and why

Abstract

Background

Disparity-reduction programs have been shown to vary in the degree to which they achieve their goal; yet the causes of these variations is rarely studied. We investigated a broad-scale program in Israel’s largest health plan, aimed at reducing disparities in socially disadvantaged groups using a composite measure of seven health and health care indicators.

Methods

A realistic evaluation was conducted to evaluate the program in 26 clinics and their associated managerial levels. First, we performed interviews with key stakeholders and an ethnographic observation of a regional meeting to derive the underlying program theory. Next, semi-structured interviews with 109 clinic teams, subregional headquarters, and regional headquarters personnel were conducted. Social network analysis was performed to derive measures of team interrelations. Perceived team effectiveness (TE) and clinic characteristics were assessed to elicit contextual characteristics. Interventions implemented by clinics were identified from interviews and coded according to the mechanisms each clinic employed. Assessment of each clinic’s performance on the seven-indicator composite measure was conducted at baseline and after 3 years. Finally, we reviewed different context-mechanism-outcome (CMO) configurations to understand what works to reduce disparity, and under what circumstances.

Results

Clinics’ inner contextual characteristics varied in both network density and perceived TE. Successful CMO configurations included 1) highly dense clinic teams having high perceived TE, only a small gap to minimize, and employing a wide range of interventions; (2) clinics with a large gap to minimize with high clinic density and high perceived TE, focusing efforts on tailoring services to their enrollees; and (3) clinics having medium to low density and perceived TE, and strong middle-management support.

Conclusions

Clinics that achieved disparity reduction had high clinic density, close ties with middle management, and tailored interventions to the unique needs of the populations they serve.

About the Authors:

Sivan Spitzer-Shohat

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Validation, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Department of Population Health, Azrieli Faculty of Medicine, Bar-Ilan University, Safed, Israel, Center for Health and Social Sciences, University of Chicago, Chicago, IL, United States of America

ORCID http://orcid.org/0000-0001-9656-0034

Efrat Shadmi

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Supervision

Affiliations Faculty of Social Welfare and Health Sciences, University of Haifa, Haifa, Israel, Clalit Research Institute, Chief Physician’s Office, Clalit Health Services, Tel Aviv, Israel

Margalit Goldfracht

† Deceased.

Roles Writing – review & editing

Affiliation: Clalit Community Division, Clalit Health Services, Tel Aviv, Israel

Calanit Key

Roles Writing – review & editing

Affiliation: Clalit Community Division, Clalit Health Services, Tel Aviv, Israel

Moshe Hoshen

Roles Data curation, Writing – review & editing

Affiliation: Clalit Research Institute, Chief Physician’s Office, Clalit Health Services, Tel Aviv, Israel

Ran D. Balicer

Roles Writing – review & editing

Affiliations Clalit Research Institute, Chief Physician’s Office, Clalit Health Services, Tel Aviv, Israel, Public Health Department, Faculty of Health Sciences, Ben-Gurion University, Beer-Sheva, Israel

Introduction

Disparity reduction efforts, focusing on minimizing differences in health and health care, have been the focus of health care systems for over 30 years[1]. Although evidence of success is accumulating[2–4], programs vary in how well they achieve their goals, with some even showing increased disparities over time[5,6]. To better understand how success is achieved, recent research calls for in-depth investigation of how programs work, evaluating the factors associated with success on a variety of outcomes, and understanding the processes that lead to attainment of disparities reduction[1,7–9].

Israel is an example of a society in which, despite the availability of national health insurance since 1995[10], gaps are evident in a wide range of indicators, such as life expectancy, infant mortality, prevalence of chronic diseases, and use of health care services[11–17]. To address these expanding gaps, in 2009 Clalit Health Services, Israel’s largest non-for-profit insurer and provider of services, covering about 54% of the population, initiated an organization-wide program to reduce gaps between low-SES and minority populations and its general member population [2]. The program targeted 55 primary care clinics serving approximately 400,000 people (10% of Clalit’s population), of mainly economically disadvantaged and minority groups, who were identified as performing poorly on a composite measure of seven health and health care indicators (the Quality Indicator Disparity Scale—QUIDS): diabetes, hypertension, and lipid control; prevention of anemia in infants; mammography screening; fecal occult blood tests; and influenza vaccinations. The aim of the program was to close the gap between target clinics and all other comparable clinics (medium to large, serving 5,000–15,000 members). Although the overall organizational goals for disparity reduction and the measurement scheme were set by the central Clalit management, the interventions formulated and their implementation strategy were developed locally at the regional, subregional, and primary-care clinic levels.

Between 2009 and 2012 a significant decrease in the gap, based on the composite quality measure, was reported[2,18]. However, clinics differed significantly in their outcomes, eliciting the need to better understand the sources of these variations[19]. To address this need, we performed a Realist Evaluation (RE), to identify ‘what works in which circumstances and for whom?’ [20]. We assessed contextual drivers, including clinic team perceptions on possessing the necessary resources and skills to succeed (perceived team effectiveness [TE]), as well as relationships among team members and with their superior managerial units. We also examined the types of interventions employed by each of the participating clinics, and the relation between the unique organizational context, the types of interventions implemented, and the attainment of the organizational disparity-reduction goal.

Methods

Setting

The program was initiated and carried out in Clalit, that operates its integrative health-care delivery system through a decentralized structure comprised of a central management and eight semi-independent geographic regional management units spread throughout the country. Each region manages its own budget, funded through capitated payment, and provides all primary and specialty care, imaging, lab, and pharmacy services to its member population. Within each region, the regional headquarters oversees 3 to 4 subregional management in charge of all the services in that subregion, including 5 to 10 primary care clinics, operating similarly to Accountable Care Organizations (ACOs) [21]. Primary care clinics are organized in teams of physicians, nurses, pharmacists, and administrative employees. Since 2002 all health care encounters in Clalit are electronically registered and data are stored and managed in a central data warehouse, through which financial, clinical, and health-care-quality performance is assessed [22].

Research strategy

We employed an RE framework, identifying the context in which each program operates, the mechanisms through which it is implemented, and its outcome, to form context-mechanism-outcome (CMO) configurations [23–26]. Data were collected using a retrospective cross-sectional mixed-methods sequential explanatory design (qualitative and quantitative)[27]. We evaluated disparity-reduction program implementation in 26 of the 55 target clinics and their associated managerial levels, which serve 217,306 enrollees in four of the organization’s eight geographic regions. Clinics were selected (from the pool of 55), to ensure representation of the characteristics of the focus clinics’ population such as rural (n = 19) and urban (n = 7) areas, and representation of patients’ ethnicity and cultural including Jewish ultra-orthodox communities (n = 2) and low-socio economic peripheral communities (n = 4) and Arab minorities (n = 20) [2]. Analysis was carried out in three phases: (1) identifying the relevant program theory through key stakeholders, (2) collecting data and analyzing program implementation, and (3) proposing CMO configurations to explain the observed variations in target clinics’ success in reducing disparity.

Phase 1: Identifying program theory of disparity reduction.

In line with the realist methodology [20], as a first step, we interviewed key informants (n = 6) from all managerial levels in the organization: head of the disparity-reduction program at Clalit headquarters, head of the organization’s quality unit, a regional headquarters director, a regional headquarters quality director, a subregional medical director, and a clinic director. Interviews focused on eliciting information about program implementation and on understanding the organizational framework for action. An ethnographic observation of a regional meeting that included inter-professional managerial-team members of target clinics: nursing, medical, administrative, and pharmacy personnel from the regional, subregional, and clinic teams, was also performed. We observed the following domains: priorities set by clinics during the presentation, the make-up of the presenting team, the type of discourse with the managerial level (e.g., defensive, open, oriented towards problem solving), and the types of solutions and directives provided by the management (e.g., focused, integrative, geared towards sustainability).

Data, including observational notes and transcriptions of interviews held with key informants, were analyzed, eliciting themes relating to both the overall program theory underpinning the disparities reduction program as well as strategies for action. Plausible strategies, that is, mid-range theories, motivating the focus of program implementation were identified. Meetings were held with study team members to discuss emerging themes, which were then refined based on feedback and discussion to strengthen validity [19,28].

Phase II: Data sources and analysis of the disparity-reduction program.

To capture the complex and dynamic nature of the disparity-reduction effort, we drew on a wide range of data sources, methods, and materials [28,29], including in-depth semi structured interviews with all stakeholders at all managerial levels, social network analysis (SNA) of data on the relationships between all actors participating in the program, the clinic teams' perceptions of their ability to implement the program (TE), data on clinic characteristics such as size and patient population served, and QUIDS scores at baseline and 3-year follow-up. The data sources are presented in Fig 1 and outlined below.

[Figure omitted. See PDF.]

Fig 1. Outline of organizational structure and data collection strategy.

d = clinic director, a physician; n = nursing director; a = administrative head; p = pharmacist.

https://doi.org/10.1371/journal.pone.0193179.g001

Qualitative data sources and analysis.

Interviews were conducted between July 2010- May 2011 by two of the authors. To elicit a wide variety of perspectives [30], we interviewed 80 inter-professional managerial-team members, including medical directors, nursing directors, heads of administration, and pharmacists (in clinics with an in-house pharmacy) in each of the 26 clinics; 19 medical and nursing directors in 10 subregional headquarters; 8 medical and nursing regional headquarter directors; and 2 quality-assurance directors. Verbal consent was obtained from interviewees.

The interview guide was adapted from the work of Chin and colleagues (2008) [31] and Wasserman and Faust (1994) [32], and then pilot-tested with key informants at five clinics. Interviews focused on (a) eliciting the perceptions of the stakeholders about what worked (or not) in achieving disparity reduction, and (b) understanding the organizational processes by probing issues such as the day-to-day relations among clinic team members, the relationship between clinic teams and their respective managerial units, and the relationship with the community of the patient population they serve. Questions also aimed to identify what interventions were implemented to address each of the seven QUIDS indicators and to explore participants’ reasoning and resources, that is, the mechanisms underpinning the strategy selected [26].

Interviews were recorded and transcribed, and a key informant was identified for each clinic. The information provided by the key informant was augmented by information from other team members and from the relevant subregional managerial personnel. Upon completing the interviews, we performed a member check by sending summaries to each clinic, to validate the information elicited and to allow interviewees the opportunity to refine or add comments [33].

We mapped the mechanisms according to the program theory identified in phase I [19]. We coded the data based on emergent themes through an iterative process of reading and re-reading the interview transcripts. We conducted a thematic analysis to identify the strategies employed and interventions implemented [34]. We then rated the scope of each type of intervention as low, medium, or high, reflecting the degree to which the selected intervention was a priority and the extent of its implementation [35]. The coding technique was pilot-tested by two of the authors (SS and ES), who independently classified data from four clinics and then discussed their findings to reach consensus.

Quantitative data collection.

Social network data were obtained through a questionnaire administered to clinic teams and subregional headquarters directors, which included a roster for identifying network ties. The roster listed the relevant organizational positions at the clinic, subregional, and regional levels. Respondents were asked to rate “To what extent do you consult with members of your organization on matters pertaining to the disparity reduction initiative?” [36]. Answers ranged from 1 (very little) to 5 (a lot). Data on the types and intensity of ties were used to generate matrices of ties between network actors (nodes) and to assess structural-network measures of density using a weighted measure (the sum values of all ties divided by the number of possible ties) of within clinic networks, clinic–subregional headquarters networks, clinic–regional headquarters networks, and overall regional networks (including all clinic, subregional, and regional teams) [32,37]. The measure of density reflects the number of actors in contact. The connection of actors in dense networks can provide social support to team members when they are implementing a new program [38,39].

Perceived Team Effectiveness (TE) was assessed using the validated team effectiveness instrument to determine whether clinic teams believed they had the necessary information, authority, and autonomy to implement the disparity-reduction program; whether they had the skills to implement the necessary changes; whether they agreed with the organization’s goal to minimize disparities and on how this should be achieved; and whether they felt supported by the organization in implementing the program S1 File [40]. A clinic composite perceived TE score was constructed based on the average rating of all participants in each of the 26 clinics. To enable the creation of CMO configurations according to perceived TE levels, we categorized clinics into tertiles as low-, medium-, and high-performing TE clinics based on the following cut points: 3.68 or below, 3.69 to 4.02, and above 4.02, respectively.

QUIDS performance data for each of the clinics, as well as contextual characteristics such as clinic size and SES indicator score rating all residential areas in Israel, from 1—low SES to 10 high SES, [41] were obtained from Clalit’s central data warehouse. We calculated the gap in QUIDS scores for each clinic versus all other comparable clinics in each region at baseline (2009) and at 3-year follow-up (2012). Disparity reduction was defined as the relative reduction in the gap in QUIDS scores between 2009 and 2012 for each participating clinic compared with the average score of all other clinics in its region. We categorized clinics according to the QUIDS gap at the beginning of the disparity-reduction program (small versus large gap) according to the following tertiles: small gap (< 0.06), medium gap (0.06–0.09), and large gap (>0.09).

Phase III: Constructing clinics’ context-mechanism-outcome configurations.

CMO configurations were constructed through an iterative process, checked against collected data, and refined [19,42]. Reduction in the gap in QUIDS scores between 2009 and 2012 for each participating clinic was assessed against contextual characteristics of the clinics and the mechanisms they employed. Configurations were identified by assessing different patterns of change and their interplay with elicited mechanisms, and by comparing clinics that attained the organizational goal with clinics that did not.

The study protocol was approved by the Committee on Human Studies, Clalit Health Services (authorization no. 172/2011).

Results

Phase I: Analysis of key stakeholders’ perspectives

Analysis of stakeholder's interviewed revealed several themes that guided their approach towards disparity reduction, including: quality improvement and organizational performance. The stakeholders emphasized the uniqueness of Clalit’s disparity-reduction program in its aim to not only monitor but create a plan of action: “Other programs aim to improve overall quality while this program targets the clinics which are the lowest-performing and most challenged. To improve overall average performance, it is usually easier to work with the early adopters, those most receptive to change. Yet, the disparity-reduction program put the spotlight on those clinics in which it is most difficult to achieve change, forcing us and them to put extra effort to improve at a faster rate than the average rate of improvement” [Head, Disparity-Reduction Program, central management].

Management focused on performance measurement, and perceived clinics as context experts responsible of tailoring the program to their patient population: “As in similar organization-wide strategies, we only ‘set the stage’ by setting the goals and providing monitoring and feedback. In order to tailor the program for unique populations, it was clear that the improvement strategy should be designed by each clinic and supported by regional headquarters” [Head, Community Quality-Improvement Unit, central management].

Analysis of Interviews with a target clinic directors as well as ethnographic observation of a regional meeting showed of an existing similar discourse to that of central management in the division of responsibilities between measurement and implementation. Clinics combined knowledge of performance data with adapting and tailoring services to the clinic’s setting: “I looked at the micro-surrounding of the clinic and translated the way we work to fit our surrounding… only after we trained the clinic team on the meaning and measurement of each indicator [could we] think how to adapt it to our patients… This is the process that I see all successful clinics have adopted… working in a two-stage process of training the team and then adapting care to the patient population served, which is the most challenging task that we have” [Clinic Director].

Analysis of the discourse elicited in this phase highlighted the organization’s focus on reducing disparities mainly through the framework of quality improvement (QI), an outcome-driven approach centering on identifying a process with less than ideal outcomes, measuring key performance attributes, and devising a plan of action [43].

Utilizing QI as the guiding program theory, we searched for common intervention strategies. QI programs focus on implementing changes at the provider and/or patient level [1]. Provider-focused strategies aim to improve the organization’s work, and they emphasize internal processes such as teamwork, delivery of services, and data management [38,44]. Patient-level interventions often focus on providing patient-centered services; most of these interventions focus on educating patients about specific chronic conditions [45–47]. Hence, we elicited two main mechanisms: (A) a focus on the provider, i.e. improving day-to-day operations through teamwork, delivery-system design (such as improved access to specialists), and promoting effective use of the organization’s patient registry of computerized electronic health records; and (B) a focus on patients, aimed at patient education and at tailoring care processes and services to the needs of the specific patient population, taking into account cultural and religious values and beliefs, performing outreach and establishing linkages with community services/leaders. The two main mechanisms identified are mechanisms implemented by clinic teams. While the provider mechanisms focuses internally on improvement by clinic teams on the way in which the day to day work is implemented, the patient focused mechanisms focuses on the way in which clinic teams’ address external patient factors that impact their day to day work.

Phase II: Analysis of target clinics’ characteristics

Context of target clinics.

Table 1 details the contextual characteristics of each of the 26 clinics. We examined the characteristics of the clinics’ patient populations (outer context) as well as the clinics’ teams (inner context). Twenty clinics served enrollees of Arab minority populations; one clinic served an ultra-orthodox Jewish community (d1). Most target clinics were large, serving over 6,000 enrollees, from disadvantaged low-SES populations. Clinics with high SES scores all had relatively high percentages of enrollees over the age of 65 (a4, c1, c2, c3).

[Figure omitted. See PDF.]

Table 1. Context: Characteristics of target clinics.

https://doi.org/10.1371/journal.pone.0193179.t001

The inner organizational characteristics of the clinics varied in terms of both network density and perceived TE. Clinics a2, c1, and d1 reported high inner clinic density as well as high levels of perceived TE. Clinics a3, a4, a8, a9, d3 and d4, on other hand, reported low inner clinic density and low perceived TE levels. Some clinics reported a mix of dense ties with subregional headquarters and low levels of TE (a5 and a9).

Mechanisms employed by target clinics.

Table 2 demonstrates the two main mechanisms—focusing on the provider and focusing on the patient—elicited from thematic analysis of the interviews. Overall, clinics implemented 454 interventions, ranging from 9 (a7) to 29 (b3) (x = 17.5 sd±5.4). Clinics also differed in the mechanism adopted: 5 clinics concentrated on improving teamwork (a5, b1, b3, b4, b5), 8 concentrated on educating or improving care processes (a2, b1, b2, b3, b4, b5, c1, d1), and 5 focused on community outreach (a2, a8, a10, c1, d1). All clinics reported medium to high use of the organization’s patient registry to design proactive patient outreach as a preferred strategy for improving performance.

[Figure omitted. See PDF.]

Table 2. Implementation of QI program by target clinics.

https://doi.org/10.1371/journal.pone.0193179.t002

Mechanism 1: Focusing on the provider: “Improving the way clinic teams work”.

Clinics guided by this mechanism centered on improving their workflow, involving three main strategies: improving team processes, changing the design of the delivery of services, and using patient registries to track performance cards and reminders.

Clinics aimed to improve team processes by allocating tasks and creating a shared team responsibility for improving processes of care. The main strategy all target clinics used was weekly team meetings to discuss ways for improving their performance on the QUIDS indicators and to perform consultations regarding care management of complex chronic patients. During each meeting, the team reviewed medical records and devised patient-centered interventions. Team members viewed the implementation of routine meetings not only as a new strategy to improve care but also as a chance to break traditional divisions of responsibilities, which often lead to working in silos (a1, a2, a3, a6, a10, b1, b3, b4, c1, d2, d5, d7, d8).

Clinics also implemented strategies aimed at improving accessibility of health care services, especially those relevant to the QUIDS indicators. For example, to increase the number of women obtaining mammography screening, a mobile screening unit was transported to large clinics that participated in the program (a1, a6, a8, c1, d2, d7, d8). Medium-sized clinics and those close to medical centers provided a shuttle from clinic to hospital for women who needed mammography or other screening tests (b1, b2, b3, a2, a8, a10, c2, d1). Another widely adopted strategy was the creation of shared-care schemes in which an ophthalmologist rotated between primary care clinics, performing fundus tests for diabetic patients (a1, a2, a5, a6, b1, b2, b5, d6).

Using Clalit’s computerized patient registry, all 26 clinics monitored their performance and identified patients for targeted intervention in all seven chosen indicators. This included identification of patients in need of screening tests or patients with poor diabetes or hypertension control. For example, clinics used the registry to identify patients in need of occult blood tests, administered testing kits to target populations, and followed up on returned screening kits.

In cases where management identified gaps in clinic teams’ knowledge when working with the organization’s computerized registry, technical support personnel were recruited to guide the teams in these processes (a6, a8, b5, c2, d2, d5, d8). For example, region A identified effective use of patient registry as a problem in all targeted clinics and set up a ‘QI school’ to train teams to retrieve data from the system, analyze performance, and devise intervention strategies.

Mechanism 2: Focusing on the patient: “Tailoring services to patients and their community”.

Clinics for which the driving mechanism was the ‘focus on the patient’ performed activities such as cultural adaption of services and identified a need to better tailor their services to the unique needs of the populations they served. They employed a wide range of patient-education interventions aimed at improving chronic-disease care. For example, clinic c1 created a culturally tailored cooking class for Arab minority diabetic patients as well as a workshop led by a nurse and a religious leader on the importance of maintaining a healthy lifestyle. Through the use of religious scripts, patients were educated about how Islam views dietary habits and exercise. Clinic a1 publicized the importance of early cancer screening in the local newspaper, including testimonials of colon cancer patients who are community members. Clinic d1 created a pamphlet for ultra-orthodox Jewish mothers on the importance of iron supplements for infants, introducing culturally tailored messages. Clinic d5 created a tailored pamphlet on the consumption of honey and its effect on diabetes control, as honey products are commonly used in traditional sweets of Arab populations.

Most target clinics implemented health-promotion and patient-education activities in the community through lectures and health-promotion fairs held in local community centers (a1, a3, a6, b1, b2, b4, c1, d1, d2, d6). Additional activities included a weekly spot on local radio to enhance knowledge and promote activities such as the importance of vaccination (Clinic a8), and referral of ultra-orthodox Jewish women to screening services at a hospital that provides care to ultra-orthodox religious populations, despite the clinic’s closer proximity to another hospital (d1). Several clinics included flu vaccinations as part of the vaccination regimen administered to Muslims embarking on their religious pilgrimage to Mecca (a8, b3, d2, d5). Others focused on educating religious leaders and administering information on the importance of vaccination during prayer times (a8, a10, b3, b5, d8).Strategies focusing on the patient also included tailoring care plans for control of diabetes or hypertension. Clinic teams crafted care plans taking into account patients unique cultural practices. For example, in many of the clinics serving diabetic Muslim patients, care plans took into account the unique practice of the month of Ramadan and how to account for control when patients fast daily and break the fast with traditional sweets. Clinic b2, for example, implemented a unique patient education and care plan around hypertension Bedouin patients to address the issue of traditional coffee drinking.

Outcome: Clinics’ QUIDS score.

Table 3 presents QUIDS scores for all clinics at baseline and 3-year follow-up. Clinics varied in the extent of the gap measured at baseline: 7 clinics had a relatively small gap (≤0.06 QUIDS points), 7 had an intermediate gap (0.06–0.09 QUIDS points), and 12 had a large gap (>0.09 QUIDS points). At follow-up, 12 clinics had closed more than 80% of their initial gap, and 7 had decreased their gap by less than a third or even slightly increased it.

[Figure omitted. See PDF.]

Table 3. Outcomes: Change in QUIDS score of target clinics.

https://doi.org/10.1371/journal.pone.0193179.t003

Phase III: Constructing context-mechanism-outcome configurations

We first categorized clinics based on the degree to which attaining the outcome (i.e., closing the gap in QUIDs score) was challenging in terms of the baseline score. Five CMO configurations were identified based on size of the initial gap, degree of attainment of the disparity-reduction goal, and types of context and mechanisms employed. Two configurations were observed for clinics with a small gap to minimize, and three were identified for target clinics with a large gap (Table 4).

[Figure omitted. See PDF.]

Table 4. Context-mechanism-outcome configurations of target clinics.

https://doi.org/10.1371/journal.pone.0193179.t004

CMO A. These clinics had a small gap at baseline and succeeded in minimizing that gap. They had highly motivated teams with strong ties among them, employed many interventions to improve care management, and tailored services to the specific needs of their patient populations (a2, b3, b4, b5) (Table 4).

CMO B. These clinics did not attain the QI disparity-reduction goal even though they had only a small gap. This was related to each clinic’s inner contextual characteristics of low clinic density, low clinic subregional density, and low perceived TE. Team members of these clinics mentioned internal problems of working together as a team, as well as a lack of organizational support. Interventions were few, focused on use of the patient registry, such as scheduled reminders, and were mostly instigated by subregional headquarters (a3, a4, c2, c3, d3, d4).

CMO C. These clinics were characterized by a large gap at baseline, which they closed. All clinics reported high levels of clinic density, medium to high clinic subregional density, and perceived themselves as capable of implementing the program. They identified the unique needs of the populations they served and tailored all their interventions, creating a better fit between services provided and their patients’ unique needs (a8, a10, b1, b2, c1, d1, d2).

CMO D. These clinics were characterized not only as having an initial large gap to minimize but also as having low clinic density and low perceived TE. Nonetheless, they differed from CMO B clinics due to their high density with subregional headquarters. Although few interventions were implemented, management’s focus and support of clinic teams during implementation of the program by holding weekly visits and meeting with clinic staff, for example, helped attain the desired goal (a5, a7, a9).

CMO E. These clinics had a large initial gap to close, and reported internal structural difficulties. All had low-medium density and perceived TE. Clinic-subregional density scores were also intermediary ranked. In this group, as compared to clinics in CMO B, clinic teams worked not only to devise a strategy for improving teamwork, but also to tailor services to meet patients’ special needs (a1, a6, d5, d6, d7, d8). These interventions, however, did not result in reduced disparity gaps.

To summarize, clinics that managed to reduce the gap and had a small initial gap to close, had high clinic density, high clinic–subregional headquarters density, and high perceived TE, these clinics implemented a high number of interventions in all areas of change focusing on changes to both provider and patient. These, however, constituted only a relatively small number of clinics (4 of 10 with a small initial gap). Clinics that managed to reduce the gap and had a large initial gap to close, either had (1) a High clinic density, medium-high subregional management density Medium-High TE and a medium-high number of interventions, yet they focused on adapting services to patients as well as developing community linkages, or (2) had Medium clinic density, high clinic–subregional management density, and high perceived TE, yet a small number of interventions implemented which were mostly instigated by management. Nonetheless, of all clinics with a large initial gap, most (10/16) managed to significantly close the gap.

Discussion

Our study focused on identifying mid-range theories and the causal linkages between context, mechanisms, and outcomes achieved in a disparity-reduction program. The first phase of the study identified QI as the program theory underpinning the disparity reduction efforts and guiding the ideas and assumptions of teams implementing the program. Understanding that QI is the guiding framework enabled us to identify the mechanisms and contexts in which the program was implemented, as well as the boundaries in which teams operated, i.e., their ability to benefit from the structure and framework of QI as well as their need to tailor their approach and tools for achieving disparity reduction given the limitations of QI. The QI approach benefited the teams by directing their actions through highlighting the measures that required improvement in their clinic's population. However, as quality indicators focus on specific elements of care, rather than how care should be delivered [48] teams were tasked with developing and culturally adapting care strategies to meet their specific population's needs. The advantages and limitations of the QI approach are reflected in the two broad mechanisms identified: (a) A focus on the provider, especially improving teamwork and service delivery design, and (b) A focus on the patient and tailoring programs to meet patient and community needs. These mechanisms explain the reasoning, actions and responses of teams to organizational program focused on achieving quality improvement disparity reduction [20].

Target clinics focused on improving how teams operate through organizational redesign as well as effective use of performance data. CMO configurations suggest that improvement of internal organizational processes, such as training clinic teams to measure and use clinical indicators, were a necessary step toward achieving disparity reduction. Similarly, Thorlby and colleagues (2008) [49] found that information systems were viewed as essential components of disparity-reduction efforts, as they allowed not only identification of existing inequities but tailoring of interventions accordingly. Nadeem and colleagues (2013) [50], in their systematic review of QI collaboratives, also report that components such as data reporting, training providers in QI techniques, and leadership involvement had the greatest impact on QI program success. Similarly, we found that all successful clinics had close ties with subregional management. Birken and colleagues (2012), [51] state that middle managers contribute to the attainment of success in QI programs by creating a bridge between the contextual factors affecting the implementation in each of their subordinate units and managing the demands of top management.

Our findings on the implementation of patient-centered interventions receive credence from a review spanning 30 years of QI disparity-reduction efforts [1], which showed that most interventions were targeted toward improving patient-centered care. Methods to trigger behavioral change centered on patient education. However, patient education was found to be a ‘necessary but not sufficient’ step. The authors conclude that organizations need to address not only the patient but the provider through interventions focused on the care-delivery system. Fiscella and Epstein (2008) [52] note that during a 15-minute visit it is impossible for physicians to properly address the needs of disadvantaged patients and that most implemented interventions need to extend the scope of clinic visits. They suggest the use of primary health-care teams centered around patients’ needs and working together as an inter-professional care-management team, as a framework for action. Similarly, we found that the implementation of weekly joint consultations for chronic-disease patients with complex care needs was associated with the attainment of disparity-reduction goals.

We also found that a focus on addressing patients’ unique sociocultural needs was associated with success. Chin and colleagues [53] found that culturally tailored interventions targeting different causes of disparities hold the most promise in effectively reducing disparities. Betancourt and colleagues (2014) [47] highlight that successful disparity-reduction efforts do not center only on language barriers but address the values, beliefs, and behaviors of cultural groups. Clinics that implemented culturally tailored interventions recognizing the unique social values and beliefs of communities, such as working together with religious leaders on the importance of lifestyle and adherence to diabetes treatment, reduced their gap and sustained this reduction over time.

Our study has several limitations. We recognize the possibility of recall bias as a threat to the validity of theory-informed data collection and note that the RE framework aims to assess the program during its implementation, refining proposed theories through participant feedback [23]. We tried to address this limitation by conducting a member check to allow respondents the opportunity to refine and give feedback on the strategies observed. A related limitation is that our data were derived from 26 target clinics and not all 55 clinics included in the project. Nonetheless, as presented in the methods section, the study clinics represented a wide range of characteristics that reflect the type of population and clinics included in the larger project. Furthermore, conducting interviews with a wide range of personnel directly involved in implementing the disparity-reduction program across a variety of urban and rural settings, and in various population groups, enabled us to examine whether the program succeeded across a wide spectrum of sites [54].

Conclusion

Our study shows that theorizing the logic of QI disparity-reduction programs and understanding how it relates to both the individuals involved and the context in which programs are implemented may help us understand not only whether a program works but also how it works, and under what circumstances disparity reduction can be achieved [55]. Our study identifies the context in which QI disparity-reduction strategies should be applied by linking identified mechanisms with program activities and observed outcomes. We show that clinics that achieved disparity reduction had highly dense teams, close ties with subregional management, and identified the unique needs of the populations they serve.

Supporting information

[Figure omitted. See PDF.]

S1 File. Questionnaire adapted from Shortell et al.docx.

https://doi.org/10.1371/journal.pone.0193179.s001

(PDF)

Citation: Spitzer-Shohat S, Shadmi E, Goldfracht M, Key C, Hoshen M, Balicer RD (2018) Evaluating an organization-wide disparity reduction program: Understanding what works for whom and why. PLoS ONE 13(3): e0193179. https://doi.org/10.1371/journal.pone.0193179

References

1. Clarke AR, Goddu AP, Nocon RS, Stock NW, Chyr LC, Akuoko JA, et al. Thirty years of disparities intervention research: what are we doing to close racial and ethnic gaps in health care? Med Care. 2013;51: 1020–1026. pmid:24128746

2. Balicer RD, Shadmi E, Lieberman N, Greenberg-Dotan S, Goldfracht M, Jana L, et al. Reducing health disparities: strategy planning and implementation in Israel’s largest health care organization. Health Serv Res. 2011;46: 1281–99. pmid:21352224

3. Peek ME, Wilkes AE, Roberson TS, Goddu AP, Nocon RS, Tang H, et al. Early lessons from an initiative on Chicago’s South Side to reduce disparities in diabetes care and outcomes. Health Aff (Millwood). Project HOPE—The People-to-People Health Foundation, Inc.; 2012;31: 177–86. pmid:22232108

4. Bharel M, Santiago ER, Forgione SN, León CK, Weinreb L. Eliminating health disparities: innovative methods to improve cervical cancer screening in a medically underserved population. Am J Public Health. 2015;105 Suppl: S438–42. pmid:25905832

5. Hicks LS, O’Malley AJ, Lieu TA, Keegan T, McNeil BJ, Guadagnoli E, et al. Impact of health disparities collaboratives on racial/ethnic and insurance disparities in US community health centers. Arch Intern Med. American Medical Association; 2010;170: 279–86. pmid:20142575

6. Hunt B, Whitman S. Black:White Health Disparities in the United States and Chicago: 1990–2010. J racial Ethn Heal disparities. 2015;2: 93–100. pmid:26863247

7. Alberti PM, Bonham AC, Kirch DG. Making Equity a Value in Value-Based Health Care. Acad Med. 2013;88. Available: http://journals.lww.com/academicmedicine/Fulltext/2013/11000/Making_Equity_a_Value_in_Value_Based_Health_Care.22.aspx

8. Dixon-Woods M, Bosk CL, Aveling EL, Goeschel CA, Pronovost PJ. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q. 2011;89: 167–205. pmid:21676020

9. Weinick RM, Hasnain-Wynia R. Quality improvement efforts under health reform: how to ensure that they help reduce disparities—not increase them. Health Aff (Millwood). 2011;30: 1837–43. pmid:21976324

10. Epstein L, Goldwag R, Isma’il S, Greenstein M RB. Reducing health inequality and health inequity in Israel: towards a national policy and action program. 2006.

11. Tarabeia J., Amitai Y., Green M., Halpern G., Blau S., Ifrah A., Rotem N and, Jaber L. Differences in Infant Mortality Rates between Jews and Arabs in Israel, 1975–2000. Isr Med Assoc J. 2004;6: 403–407. pmid:15274530

12. Chernichovsky D, Anson J. The Jewish-Arab divide in life expectancy in Israel. Econ Hum Biol. 2005;3: 123–37. pmid:15722265

13. Ministry of Health. 60 Years of Health in Israel. Number 315 (in Hebrew). [Internet]. 2008. Available: http://www.health.gov.il/PublicationsFiles/ICDC_315.pdf

14. Baron-Epel O, Garty N, Green MS. Inequalities in use of health services among Jews and Arabs in Israel. Health Serv Res. 2007;42: 1008–19. pmid:17489901

15. Tandeter H, Masandilov I, Kemerly I, Biderman A. Ethnic differences in preventive medicine: the example of Jewish Ethiopian women in Israel. Isr Med Assoc J. 2007;9: 452–6. pmid:17642393

16. Benyamini Y, Blumstein T, Boyko V, Lerner-Geva L. Cultural and educational disparities in the use of primary and preventive health care services among midlife women in Israel. Womens Health Issues. 18: 257–66. pmid:18329896

17. Na’amnih W, Muhsen K, Tarabeia J, Saabneh A, Green MS. Trends in the gap in life expectancy between Arabs and Jews in Israel between 1975 and 2004. Int J Epidemiol. Oxford University Press; 2010;39: 1324–32. pmid:20534651

18. Balicer RD, Hoshen M, Cohen-Stavi C, Shohat-Spitzer S, Kay C, Bitterman H, et al. Sustained Reduction in Health Disparities Achieved through Targeted Quality Improvement: One-Year Follow-up on a Three-Year Intervention. Health Serv Res. Clalit Health Services, Tel Aviv, Israel.; 2015; pmid:25787874

19. Schierhout G, Hains J, Si D, Kennedy C, Cox R, Kwedza R, et al. Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change. Implement Sci. 2013;8: 119. pmid:24098940

20. Pawson Ray TN. Realistic Evaluation. SAGE Publications; 1997.

21. Filc D. Circles of Exclusion: Obstacles in Access to Health Care Services in Israel. Int J Heal Serv. SAGE Publications; 2010;40: 699–717. pmid:21058539

22. Cohen A D, Driher J, Regev-Rosenberg S, Jacobson O, Liberman N, Goldfracht M BRD. The Quality Indicator Program in Clalit Health Services: The First Decade. Harefua (in Hebrew). 2010;149: 204–209.

23. Pawson R. Evidence-Based Policy A Realist Perspective. SAGE Publications; 2006.

24. Pawson R, Manzano-Santaella A. A realist diagnostic workshop. Evaluation. 2012;18: 176–191.

25. Greenhalgh T, Wong G, Jagosh J, Greenhalgh J, Manzano A, Westhorp G, et al. Protocol—the RAMESES II study: developing guidance and reporting standards for realist evaluation. BMJ Open. 2015;5: e008567. pmid:26238395

26. Dalkin SM, Greenhalgh J, Jones D, Cunningham B, Lhussier M. What’s in a mechanism? Development of a key concept in realist evaluation. Implement Sci. BioMed Central Ltd; 2015;10: 49. pmid:25885787

27. Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. SAGE Publications; 2011.

28. Greenhalgh T, Humphrey C, Hughes J, Macfarlane F, Butler C, Pawson R. How do you modernize a health service? A realist evaluation of whole-scale transformation in london. Milbank Q. Blackwell Publishing Inc; 2009;87: 391–416. pmid:19523123

29. Doi L, Jepson R, Hardie S. Realist evaluation of an enhanced health visiting programme. Harris F, editor. PLoS One. Public Library of Science; 2017;12: e0180569. pmid:28672013

30. Yin RK. Case Study Research: Design and Methods [Internet]. Bickman L, Rog DJ, editors. Essential guide to qualitative methods in organizational research. Sage Publications; 2009.

31. Chin MH, Kirchhoff AC, Schlotthauer AE, Graber JE, Brown SE, Rimington A, et al. Sustaining quality improvement in community health centers: perceptions of leaders and staff. J Ambul Care Manage. 2008;31: 319–329. pmid:18806592

32. Wasserman S, Faust K. Social network analysis: methods and applications [Internet]. American Ethnologist. 1994.

33. Sandelowski M. Member check. In: Given L, editor. The Sage Encyclopedia of Qualitative Research Methods. SAGE Publications; 2008.

34. Stewart M, Makwarimba E, Barnfather A, Letourneau N, Neufeld A. Researching reducing health disparities: mixed-methods approaches. Soc Sci Med. 2008;66: 1406–17. pmid:18248867

35. Green J, Willis K, Hughes E, Small R, Welch N, Gibbs L, et al. Generating best evidence from qualitative research: the role of data analysis. Aust N Z J Public Health. 2007;31: 545–50. pmid:18081575

36. Zohar D, Tenne-Gazit O. Transformational leadership and group interaction as climate antecedents: a social network analysis. J Appl Psychol. 2008;93: 744–757. pmid:18642981

37. Scott J. Social Network Analysis. SAGE Publications; 2012.

38. Meltzer D, Chung J, Khalili P, Marlow E, Arora V, Schumock G, et al. Exploring the use of social network methods in designing healthcare quality improvement teams. Soc Sci Med. Elsevier Ltd; 2010;71: 1119–30. pmid:20674116

39. Valente TW, Palinkas LA, Czaja S, Chu K-H, Brown CH, Hipp P. Social Network Analysis for Program Implementation. Moore S, editor. PLoS One. Hampton Press; 2015;10: e0131712. pmid:26110842

40. Shortell SM, Marsteller JA, Lin M, Pearson ML, Wu S-Y, Mendel P, et al. The role of perceived team effectiveness in improving chronic illness care. Med Care. 2004;42: 1040–8. pmid:15586830

41. CBS. Characterization and Classification of Geographical Units by the Socio-Economic Level of the Population 2008. Jerusalem, Israel.; 2013.

42. Jagosh J, Bush PL, Salsberg J, Macaulay AC, Greenhalgh T, Wong G, et al. A realist evaluation of community-based participatory research: partnership synergy, trust building and related ripple effects. BMC Public Health. BioMed Central; 2015;15: 725. pmid:26223523

43. Kleinman LC, Dougherty D. Assessing quality improvement in health care: theory for practice. Pediatrics. 2013;131 Suppl: S110–9. pmid:23457146

44. Chin MH. Quality improvement implementation and disparities: the case of the health disparities collaboratives. Med Care. 2010;48: 668–75. pmid:20613665

45. Chin MH, Walters AE, Cook SC, Huang ES. Interventions to reduce racial and ethnic disparities in health care. Med Care Res Rev. 2007;64: 7S–28S. pmid:17881624

46. Lurie N, Fremont A, Somers SA, Coltin K, Gelzer A, Johnson R, et al. The National Health Plan Collaborative to Reduce Disparities and Improve Quality. Joint Commission Resources;

47. Betancourt JR, Corbett J, Bondaryk MR. Addressing disparities and achieving equity: cultural competence, ethics, and health-care transformation. Chest. American College of Chest Physicians; 2014;145: 143–8. pmid:24394825

48. Rubin HR, Pronovost P, Diette GB. The advantages and disadvantages of process-based measures of health care quality. Int J Qual Heal Care. Oxford University Press; 2001;13: 469–474.

49. Thorlby R, Jorgensen S, Siegel B, Ayanian JZ. How health care organizations are using data on patients’ race and ethnicity to improve quality of care. Milbank Q. 2011;89: 226–255. pmid:21676022

50. Nadeem E, Olin SS, Hill LC, Hoagwood KE, Horwitz SM. Understanding the components of quality improvement collaboratives: a systematic literature review. Milbank Q. 2013;91: 354–94. pmid:23758514

51. Birken SA, Lee S-YD, Weiner BJ. Uncovering middle managers’ role in healthcare innovation implementation. Implement Sci. 2012;7: 28. pmid:22472001

52. Fiscella K, Epstein RM. So much to do, so little time: care for the socially disadvantaged and the 15-minute visit. Arch Intern Med. NIH Public Access; 2008;168: 1843–52. pmid:18809810

53. Chin MH, Clarke AR, Nocon RS, Casey AA, Goddu AP, Keesecker NM, et al. A roadmap and best practices for organizations to reduce racial and ethnic disparities in health care. J Gen Intern Med. 2012;27: 992–1000. pmid:22798211

54. Manzano-Santaella A. A realistic evaluation of fines for hospital discharges: Incorporating the history of programme evaluations in the analysis. Evaluation. 2011;17: 21–36.

55. Lacouture A, Breton E, Guichard A, Ridde V. The concept of mechanism from a realist approach: a scoping review to facilitate its operationalization in public health program evaluation. Implement Sci. 2015;10: 153. pmid:26519291