An integrated hospital-district performance evaluation for communicable diseases in low-and middle-income countries: Evidence from a pilot in three sub-Saharan countries

Abstract

Introduction

The last two decades saw an extensive effort to design, develop and implement integrated and multidimensional healthcare evaluation systems in high-income countries. However, in low- and middle-income countries, few experiences of such systems implementation have been reported in the scientific literature. We developed and piloted an innovative evaluation tool to assess the performance of health services provision for communicable diseases in three sub-Saharan African countries.

Material and methods

A total of 42 indicators, 14 per each communicable disease care pathway, were developed. A sub-set of 23 indicators was included in the evaluation process. The communicable diseases care pathways were developed for Tuberculosis, Gastroenteritis, and HIV/AIDS, including indicators grouped in four care phases: prevention (or screening), diagnosis, treatment, and outcome. All indicators were calculated for the period 2017–2019, while performance evaluation was performed for the year 2019. The analysis involved four health districts and their relative hospitals in Ethiopia, Tanzania, and Uganda.

Results

Substantial variability was observed over time and across the four different districts. In the Tuberculosis pathway, the majority of indicators scored below the standards and below-average performance was mainly reported for prevention and diagnosis phases. Along the Gastroenteritis pathway, excellent performance was instead evaluated for most indicators and the highest scores were reported in prevention and treatment phases. The HIV/AIDS pathway indicators related to screening and outcome phases were below the average score, while good or excellent performance was registered within the treatment phase.

Conclusions

The bottom-up approach and stakeholders’ engagement increased local ownership of the process and the likelihood that findings will inform health services performance and quality of care. Despite the intrinsic limitations of data sources, this framework may contribute to promoting good governance, performance evaluation, outcomes measurement and accountability in settings characterised by multiple healthcare service providers.

About the Authors:

Lara Tavoschi

Contributed equally to this work with: Lara Tavoschi, Paolo Belardi

Roles Conceptualization, Methodology, Supervision, Validation, Writing – original draft, Writing – review & editing

Affiliation: Department of Translational Research and New Technologies in Medicine and Surgery, University of Pisa, Pisa, Italy

Paolo Belardi

Contributed equally to this work with: Lara Tavoschi, Paolo Belardi

Roles Conceptualization, Data curation, Methodology, Validation, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation: Management and Healthcare Laboratory, Institute of Management and Department EMbeDS, Sant’Anna School of Advanced Studies, Pisa, Italy

ORCID logo https://orcid.org/0000-0002-8041-8989

Sara Mazzilli

Roles Data curation, Writing – original draft

Affiliation: Scuola Normale Superiore, Pisa, Italy

Fabio Manenti

Roles Conceptualization, Validation, Writing – review & editing

Affiliation: Doctors with Africa CUAMM, Padova, Italy

Giampietro Pellizzer

Roles Conceptualization, Validation, Writing – review & editing

Affiliation: Doctors with Africa CUAMM, Padova, Italy

Desalegn Abebe

Roles Data curation, Validation

Affiliation: Doctors with Africa CUAMM, St. Luke Wolisso Hospital/Wolisso Catchment Area, Wolisso, Ethiopia

Gaetano Azzimonti

Roles Data curation, Validation

Affiliation: Doctors with Africa CUAMM, Tosamaganga District Designated Hospital/Iringa District Council, Iringa, Tanzania

John Bosco Nsubuga

Roles Data curation, Validation

Affiliation: Doctors with Africa CUAMM, St. Kizito Matany Hospital/Napak District, Matany, Uganda

Giovanni Dall’Oglio

Roles Data curation, Validation

Affiliation: Doctors with Africa CUAMM, Pope John XXIII Aber Hospital/Oyam District, Gulu, Uganda

Milena Vainieri

Roles Conceptualization, Validation, Writing – review & editing

Affiliation: Management and Healthcare Laboratory, Institute of Management and Department EMbeDS, Sant’Anna School of Advanced Studies, Pisa, Italy

Introduction

Despite steady global improvements, the burden of disease due to communicable diseases (CDs) remains high in certain regions of the world, including sub-Saharan Africa. Lower respiratory infections, diarrhoeal diseases and sexually transmitted infections (including HIV) still rank high among the causes of morbidity and death [1]. Dedicated public health strategies have been designed and rolled out over the decades to tackle CDs in sub-Saharan Africa, often engaging multiple partners for their implementation. Global initiatives such as the Global Fund to Fight AIDS, Tuberculosis (TB) and Malaria and Stop-TB, have greatly accelerated progress by fostering multi-stakeholders’ collaborations and resources mobilization [2]. Healthcare provision in certain areas of sub-Saharan Africa relies also on the presence of external agencies and entities such as non-governmental organizations (NGOs) [3, 4].

The heterogeneity of the healthcare services (HS) scenario poses additional challenges for performance assessment, especially in countries where national monitoring systems are sub-optimal. Performance evaluation of healthcare provision is extremely relevant to i) assess the impact of different stakeholders’ efforts and contributions towards the achievement of health goals; ii) monitor activities and plan policy-related initiatives; iii) set and promote quality improvement actions at national/local level; and iv) ensure accountability towards national health institutions and donors [5, 6].

In recent years, an extensive effort to design, develop and implement Performance Evaluation Systems (PESs) in healthcare have been made by institutions in single high-income countries as well as by international agencies [6–8]. The added value of PES rests on the integrated approach, taking into consideration multiple dimensions and indicators related to efficiency, structure, process, quality of care, appropriateness, and equity [9] as well as the interests of several stakeholders in the healthcare system, from a population-based perspective [10]. Yet, in low- and middle-income countries (LMICs), few experiences of PESs implementation have been reported in the scientific literature [11, 12]. Shortage of human and technical resources, unavailability and unreliability of data, lack of culture of data-driven decision making are all coexisting factors that hamper the set-up and the maintenance of coherent and integrated PES in LMICs [13–15]. Additionally, a larger body of evidence relates to the application of monitoring and evaluation frameworks of global health strategies, which usually have a vertical approach and are geared towards the assessment of progresses towards global goals rather than towards the assessment of performance [16, 17]. In this scenario, quality of health data on CDs, as compared to other dimensions or health conditions, may have benefitted from more intense and coordinated efforts along the past decades, targeting LMICs and the sub-Saharan region in particular.

The present study aimed at investigating the feasibility of using a bottom-up and integrated PES, and at assessing the performance of health services provision related to CDs (PES-CDs) in three selected sub-Saharan countries. For this purpose, we adapted and tailored the framework developed to assess Italian regional healthcare systems performance through an iterative multi-stakeholders process [9], engaging local HSs, academia, and an Italian-based non-governmental organization (NGO) delivering health care in the target countries.

Material and methods

Country/Setting brief description: Similarities and differences

The analysis involved the following four hospitals and their relative health districts: i) St.Luke—Wolisso Hospital (Wolisso) and five “Woredas” in the region of Oromia in Ethiopia (Wolisso Area); ii) Tosamaganga District Designated Hospital (Tosamaganga) and the Iringa District Council (Iringa District) in the region of Iringa in Tanzania; iii) St. Kizito—Matany Hospital (Matany) and the Napak District and iv) Pope John XXIII—Aber Hospital (Aber) and the Oyam District in the northern region in Uganda, as shown in Fig 1.

[Figure omitted. See PDF.]

Fig 1. Analysed settings.

List of the four analysed hospitals and their respective health districts. Reprinted from Doctors with Africa CUAMM under a CC BY license, with permission from PLOS ONE, original copyright 2021.

https://doi.org/10.1371/journal.pone.0266225.g001

The details of the analysed hospitals and their relative health districts are included in the supporting information (S1 Table).

These settings differ not only in terms of population served, the surface area covered, and, consequently, population density, but also with respect to some relevant and intrinsic factors, which include different environmental characteristics, epidemiological priorities and issues, organizational and governance models, levels of development of transport, energy, and ICT infrastructures.

Despite such differences, they were selected because of some similar characteristics inherent to all four contexts, which are related to: i) the organization of healthcare delivery across levels of care; ii) the institutional setting of the hospitals, i.e. private, faith-based and not for profit; and iii) the funding model through which hospitals are financed. Most importantly, in all these contexts clinical and administrative activities are supported by the same NGO, Doctors with Africa CUAMM (CUAMM) which contributed to strengthening health systems in LMICs through the allocation of financial and human resources, including expatriate professionals, for several decades.

Data sources and analyses

We sourced health-related data on TB, Gastroenteritis and HIV/AIDS for the years 2017–2019. Hospital level metrics were derived from health and administrative registers, whereas district level metrics were extracted from the District Health Information Systems (DHISs) in each context of interest. In all four hospitals, TB, Gastroenteritis, and HIV/AIDS related-data were retrieved from digital and paper-based administrative department registers, which include laboratory, pharmacy, outpatient (e.g. ART clinic, TB clinic), and inpatient departments records. At health district level, data were electronically extracted from specific DHIS registers provided by the Ministry of Health of each country involved in this study. For example, TB data related to the two Ugandan sites were sourced from the Ugandan DHIS in the register 106a:3.1. Data sources are further detailed in Table 1.

[Figure omitted. See PDF.]

Table 1. List of the indicators included into the CDs care pathways: Tuberculosis, gastroenteritis and HIV/AIDS.

https://doi.org/10.1371/journal.pone.0266225.t001

Data extraction was carried out by local staff in each study context. Disease-specific datasets were created by extracting aggregated data on pre-defined variables (Table 1) from existing databases into excel spreadsheets and subsequently elaborated. Data were aggregated per variable of interest and per year of study. Disease-specific datasets were successively shared with Italian partners for analyses validation purposes. Data analyses and relative visual representations were run using SAS (Statistical Analysis System) version 9.4.

CDs care pathways development

This pilot study represents the result of constructive research carried out by two Italian academic research centres on healthcare economics and management, public health in collaboration with CUAMM. The study protocol/research approach used has been previously described elsewhere [Belardi et al., under review].

A multidisciplinary team elaborated the list of indicators populating the PES-CDs and a total number of 42 indicators, 14 for each specific CD care pathway, were selected. Additionally, 20 indicators were calculated at hospital level and 22 at health district level (Table 1).

Indicators were derived from international reference sources, including WHO relevant global monitoring frameworks [18–23]. Reference indicators were revised and adapted based on local needs or to data availability at hospital and health district levels. The final list of indicators with reference sources is presented in Table 1.

All the indicators were calculated for the years 2017–2019. For 16 indicators (three for TB, seven for Gastroenteritis, six for HIV/AIDS) data sources were not available for at least in one setting. Subsequently, each of the indicators calculated was assessed according to five dimensions [24], namely: relevance (is the indicator appropriate in relation to the peculiarities of the context analysed?); validity (is the indicator compliant with the purposes for which has been defined?); reliability (is the data and its sources authentic, solid, and reliable?); interpretability (does the indicator provide a univocal indication on how to interpret the data?); feasibility (can the indicator be calculated by using the existent information flows?; a reference standard is available?; is the reference standard appropriate for all contexts studied?). Using a consensus-building approach, a sub-set of 23 indicators per each disease-specific PES-CD was derived, covering all phases of care (prevention or screening, diagnosis, treatment and outcome) (Table 2). Evaluation was performed for the year 2019.

[Figure omitted. See PDF.]

Table 2. Care pathways’ indicators.

https://doi.org/10.1371/journal.pone.0266225.t002

The evaluation standards were determined based on: i) international and global targets, when available; ii) on the guidelines already implemented by the Italian Inter‐Regional Performance Evaluation System (IRPES) [9, 25]; or iii) on benchmarking assessment of indicators’ values statistical distribution (i.e. by applying 0 to 5 scores to five bands that consider the statistical distribution of indicators values).

The indicators were evaluated using the approach already adopted for the development of the IRPES in Italy [9, 25], and subsequently applied in some OECD countries [26–28]. To provide an integrated and continuous view of performance across different settings and providers, each care pathway was represented by the music stave (the “stave”) [9]. The staves illustrate strengths and weaknesses characterizing patient’s pathway along the continuum of care.

Results

Overall, the 2017–2019 trend was analysed for the 42 indicators. Substantial variability was observed over time and across the four different settings (Fig 5 and supporting information (S1 File)).

Here we present the performance results of the three staves (Figs 2–4) and one selected indicator for each stave (Fig 5).

[Figure omitted. See PDF.]

Fig 2. Tuberculosis care pathway.

Performance of care pathways related to tuberculosis.

https://doi.org/10.1371/journal.pone.0266225.g002

[Figure omitted. See PDF.]

Fig 3. Gastroenteritis care pathway.

Performance of care pathways related to gastroenteritis.

https://doi.org/10.1371/journal.pone.0266225.g003

[Figure omitted. See PDF.]

Fig 4. HIV/AIDS care pathway.

Performance of care pathways related to HIV/AIDS.

https://doi.org/10.1371/journal.pone.0266225.g004

[Figure omitted. See PDF.]

Fig 5. One selected indicator per each stave.

Three-year trend indicators (panels on the left), and their respective evaluated scores for 2019 (panel on the right), one for each CD pathway.

https://doi.org/10.1371/journal.pone.0266225.g005

Stave and indicators related to the TB care pathway performance

Fig 2 shows the stave related to the TB care pathway performance evaluation for the four geographical areas considered in this study.

More particularly, in Wolisso Area, the indicators evaluating the TB prevention phase scored on the orange band, while the ones belonging to the diagnosis phase from red to yellow. Average performance was also observed for the indicator evaluating the treatment phase, while the outcome phase included indicators with scores ranging from poor to excellent performance.

In Iringa District, the performance of the indicator assessing the percentage of isoniazide preventive therapy (IPT) (IDPT01) scored on the red band, while the performance of the percentage of TB patients who underwent an HIV screening (IDPT02) scored excellent. Additionally, the indicators included in the diagnosis phase scored very poor, whereas an excellent performance was registered for the indicator evaluating the treatment phase. Finally, an ups and downs trend emerged by observing the outcome indicators.

With reference to Napak District, the stave reported a very poor or poor performance for all the evaluated indicators along the four care pathway phases.

Finally, in Oyam District, the performance of the two indicators regarding the prevention phase scored very poor and poor, respectively. Higher performance scores were registered, instead, along the diagnosis and treatment phases, with a negative deterioration in both indicators of the outcome phase, which scored on the red band.

Fig 3 reports the three-year trend of the percentage of TB patients who completed the treatment in the reference year (indicator IDPT10, outcome phase). The target of 90% was set based on the WHO guidelines [29]. The indicator values ranged from 38% in the Oyam District to 93% in the Wolisso Area and scored 45% and 85% in Napak and Iringa Districts respectively. For Napak District, this observation was consistent with the other indicators along the TB care pathway. The highest percentage of patients who interrupted the treatment (S1 Fig, IDPT12) and those who resulted positive to the Xpert rifampicin-resistance test (S1 Fig, IDPT06.1) were in fact reported in Napak District.

Stave and indicators related to the gastroenteritis care pathway performance

From the analysis of the stave representing the gastroenteritis care pathway evaluation (Fig 3), it emerged that in Wolisso Area all the indicators related to the prevention phase showed excellent performance, with the exclusion of the indicator IDPD05, i.e. average number of toilets in OPD per number of rooms, which scored on the red band. Moreover, the performance of the indicators included in the diagnosis and outcome phases were registered with average and good scores, respectively. Indicators evaluating the treatment phase were not available for this setting.

With respect to Iringa District, the prevention phase showed heterogeneous results, with indicators scoring from poor to excellent performance. The indicator evaluating the diagnosis phase, i.e. the percentage of discharged patients for diarrhoea and gastroenteritis at hospital level (IDPD10), performed very well, while the one belonging to the outcome phase, namely the percentage of deaths with a diagnosis of gastroenteritis (IDPD14), scored on the orange band. Indicators evaluating the treatment phase were not available for this setting.

As in the previous stave, in Napak District the prevention phase showed diversified results in terms of performance scores, from excellent (indicators B7.9 and IDPD04) to average (IDPD02) and very poor (IDPD05). Moreover, the indicator evaluating the diagnosis phase performed poor, while higher performance scores were registered for the treatment and outcome phases.

With reference to Oyam District, the stave displayed excellent performance, except for the indicator regarding vaccination coverage for rota virus (B7.9). Additionally, the indicator related to the diagnosis phase scored poorly, whereas the indicators belonging to the other two phases showed a very good or excellent performance.

The histogram in Fig 5 shows the three-year trend of the percentage of deaths with a diagnosis of gastroenteritis and diarrhoea among hospitalized patients aged less than five years (indicator IDPD14, outcome phase). The target of 0,4% was set based on data assessment in benchmarking among the hospitals analysed. In the three-year period examined a fluctuating trend emerged. Except for Tosamaganga, the indicators values were homogeneous among districts. When comparing this indicator with the percentage of patients diagnosed with severe dehydration (S1 Fig, IDPD09), we observe some discrepancies regarding Tosamaganga. In 2019, when the highest death rate (2.0%) was reported, the hospital registered the lowest percentage of severe cases diagnosed (2.5%). Furthermore, Matany and Wolisso showed similar death rates, but widely different percentages of severe cases.

Stave and indicators related to the HIV/AIDS care pathway performance

The stave representing the HIV/AIDS care pathway (Fig 4) did not include indicators assessing primary prevention and diagnosis activities.

In Wolisso area, the performance of all evaluated indicators belonging to the screening and treatment phases scored very poor, with the exception of the percentage of HIV positive cases undergoing the TB screening (CPHIV03) that had an excellent score. Data on outcome measures were not available.

In Iringa District, the three indicators related to the screening phase scored average to excellent. Excellent performance results emerged from indicators belonging to the treatment phase, while those indicators evaluating treatment outcomes scored on the red band.

With respect to Napak District, the three indicators related to the screening phase scored poor to excellent, the indicators evaluating the treatment phase scored good to excellent, and those related to outcomes scored on the low and average bands.

In Oyam District, the screening phase showed heterogeneous results, with performance scores ranging from poor to good. Additionally, the two indicators evaluating the treatment phase (CPHIV07 and CPHIV08) scored on the dark green and red bands respectively, as well as those evaluating treatment outcomes (CPHIV10 and CPHIV11). Fig 5 illustrates the three-year trend of the HIV coverage rate of HIV therapy among residents at residence level (indicator CPHIV08, treatment phase). The denominator consists of the estimation of HIV prevalence among residents in the areas considered. The target of 95% was fixed according to WHO standard [30]. The graph shows that in the Napak and Oyam Districts almost all the estimation of HIV positive residents were enrolled on ART therapy over three years, while in the Wolisso Area and Oyam District this percentage was under 50%.

Discussion

In LMICs, performance evaluation tools are often used for reporting on achievement against international targets, providing baseline assessments under specific monitoring programmes or related to definite project goals, and ensuring accountability to donors [16, 17]. Indeed, these frameworks usually imply top-down approaches intended to evaluate outcomes at macro or project level. Here, we described the pilot experience in developing and implementing a PES-CDs in three sub-Saharan African countries using a bottom-up approach.

Indeed, the PES-CDs framework was developed through a north-south cooperation involving a NGO, two research centres and local health authorities and providers. This multisectoral cooperation benefitted from the commitment of the NGO, local stakeholders and partners to strengthen health systems activities using a data-driven and evidence-based approach and represented a fundamental condition for the development of the appropriate tool. The NGO’s knowledge of the institutional and epidemiological contexts allowed the design of the system rapidly and facilitated the process of interaction and integration with local communities and professionals. Local stakeholders’ involvement contributed to raise awareness and foster a common understanding of the jointly conceived PES-CDs, to ensure high degree of participation during all phases of development and to engage a multidisciplinary team. Moreover, this collaboration would not have been possible without the scientific contribution of academia, whose know-how guaranteed the application of rigorous methodology to PES-CDs development and data analyses taking advantage of previous experience in other healthcare systems.

The innovative contribution of this study lies in the development of a set of indicators aimed at measuring and evaluating the performance of the patient pathway along the continuum of care within the local healthcare system. Since HSs in each geographical area involve a plurality of organizations, the design of an evaluation framework that integrates different providers and their respective perspectives becomes fundamental. The PES-CDs added value relied in the assessment of the capacity of local healthcare systems to perform along the continuum of care and through the integrations of different organizations [31]. In other words, the focus of this evaluation shifted from an organization-centred approach to a patient-focused perspective [32], thus highlighting how local healthcare systems are capable of creating value for their reference populations [9, 10].

In our experience, the use of the stave as an intuitive and effective visual representation of the results proved to be a successful approach to highlight strengths and weaknesses along the patients’ care pathway. In particular, it provided policy makers and healthcare managers with an integrated and continuous view of the performance across different healthcare settings [9].

In particular, from the TB stave analysis emerged that both TB primary (IDPT01) and secondary prevention (IDPT02) could be greatly improved in all the contexts considered, even if secondary prevention performed slightly better. When assessing outcomes, the percentage of cured patients (IDPT09) scored differently across settings, and appeared not to be related to the diagnostic or treatment performance, but rather to the rate of drop out. For example, in Oyam District, where the indicators belonging to the diagnosis and treatment phases scored high, IDPT09 scored very poor. This could be attributed to the still low treatment adherence rates due to the lack of patients awareness of the importance to complete the treatment [33]. Napak District scored poorly along the whole TB pathway, registering a low rate of treatment completion, thus highlighting the challenges in ensuring adherence to treatment in an area of predominantly nomadic population. On the contrary, Iringa and Wolisso Districts registered the highest percentages of cured TB patients suggesting a good service organization and patient’s follow-up system. Appropriate and locally tailored strategies to increase treatment compliance are necessary to ensure a wider proportion of treated patients resolve the infection [34].

The Gastroenteritis care pathway was least complete due to data unavailability at both health district and hospital levels. A wide variation was observed across different settings, especially concerning the vaccination coverage for rota virus (B7.9). Indeed, such remarkable differences may be explained by the difficulty of complying with the international target (98% coverage), which is particularly challenging in these settings due to many reasons (i.e. interruption of the vaccine cold chain, natural events as well as duration of raining seasons) [35]. Information on treatment and disease outcomes was less consistent and not always available. The capacity of diagnosing and defining severe cases and their attributable deaths were not uniform across different contexts. The lack of reliability in defining causes of death is a well-known phenomenon in LMICs [36]. For example, deaths with a diagnosis of gastroenteritis and diarrhoea among hospitalized children aged <5 ranged from very poor to excellent and were not related to treatment performances. Finally, Hygiene and sanitation prevention measures met the WHO standards when implemented at the hospital level, while scored worse at the residential level, with important implications in terms of investment beyond the healthcare sector.

The HIV care pathway benefitted from a robust tradition of regional and global monitoring and vertical investments [37]. Performance indicators related to screening rarely achieved the international standards. Moreover, it was likely to have an under-reporting of HIV cases in the contexts analysed. This may have affected the estimation of treatment performance indicators and, particularly, the coverage rate of the therapy (CPHIV08), which was calculated using HIV prevalence estimates. As a consequence, the assessment of single indicators, as compared to the stave, may not provide an accurate and complete understanding of HSs provision gaps. Outcomes indicators also were affected by local practices and data availability, resulting in low accuracy and validity of the data. The most relevant example was perhaps the indicator measuring proportion of patients undergoing ART therapy and with suppression of viremia (CPHIV11) which scored from excellent to very poor across the different settings, but was likely to be severely affected by the fact that proportion of patients tested for HIV viral load during treatment scored very poorly in all the reference hospitals. This might be a very relevant finding considering that suppression of viral load is one of the key global indicators to assess country performance against WHO goals for HIV control [38].

By applying the PES-CDs to four settings located in three different countries, we tested the tool’s scalability and replicability in diversified contexts, building on previous experiences [39, 40]. This supra-national benchmarking exercise was made possible by the adoption of targets and standards defined and already in use at international level. Consequently, based on our experience, the PES-CDs indicators and graphical representation could be used by other organizations providing HS in LMICs at various levels, regardless of the epidemiological context or the organizational model.

As in high-income countries, where the HSs provision heterogeneity has been studied for over 40 years [41], in LMICs unwarranted clinical variation is also common. Our results highlighted substantial variability across all care pathway’s phases and between the four settings by mean using standardised indicators and targets. The relative weight of the underlying determinants will need to be further analysed with local stakeholders.

Health professionals’ motivation and engagement in PES-CDs may also vary as different monitoring activities (e.g. data capturing, collection, analysis, interpretation) are not always prioritized in the daily routine and often derive from top-down initiatives [42, 43]. Therefore, the commitment to capturing data of sufficient quality and reliability is often limited and clinicians are rarely involved in or responsible for the use of PES-CDs data for monitoring or planning improvements. In this scenario, the PES-CDs promoted data sharing between local healthcare system providers in the study settings and facilitated the development of local competencies in data collection, analysis and interpretation, including problem-solving. In addition, by using data obtained through hospitals and health districts’ information systems, the PES-CDs seeks to foster synergies between existing public information sources.

For all these reasons, the main aim of the PES-CDs remains its use at local level (e.g. hospital level, district level) to boost performance evaluation over time and to promote and guide hospital/district internal quality improvement efforts.

Over and above data handling limitations already mentioned, this study constituted a pilot experience and the extent of the impact on the local settings has not been evaluated yet. Future rounds of data collection, analyses and sharing with local stakeholders are planned in the coming years with the aim of turning PES-CDs into a routine activity. Further developments of the PES-CD may also be considered to assess, among others, usefulness and impact on health system financing as well as the inclusion of additional measures to investigate the dimension of equity in healthcare access.

Conclusion

We described an innovative approach to develop and roll-out a PES-CDs of health services provision along three CDs care pathways in three sub-Saharan African countries through multidisciplinary and north-south collaboration. Despite the intrinsic limitations of data sources, our results demonstrated that this system has great potential to strengthen the culture of data collection and monitoring at local level, ultimately fostering data-driven policy making and planning in healthcare. The bottom-up approach and stakeholders’ engagement increased the likelihood of local ownership of the process and engagement to implement changes needed to improve HS performance and quality of care. In health systems characterized by multiple actors (governmental entities, private funders and providers), the proposed PES-CDs may contribute to achieve good governance in HSs provision, by stimulating performance evaluation, outcomes measurement and accountability.

Supporting information

S1 Fig. Trend and evaluated indicators related to the three CDs care pathways.

https://doi.org/10.1371/journal.pone.0266225.s001

(PDF)

S1 Table. List of the analysed hospitals and their relative health districts or catchment area.

https://doi.org/10.1371/journal.pone.0266225.s002

(PDF)

S1 File. Link to the report “Performance Evaluation System of hospital and health districts in Ethiopia, Uganda and Tanzania”.

https://doi.org/10.1371/journal.pone.0266225.s003

(PDF)

S2 File. Data elements related to the indicators listed in Table 2.

https://doi.org/10.1371/journal.pone.0266225.s004

(PDF)

Acknowledgments

The authors would also like to thank all the professionals of the four hospitals and health districts for their precious work on data extraction and collection.

Wolisso Catchment Area: Stefano Vicentini, Stefano Parlamento, Yonas Desta, Ettore Boles. Iringa District Council: Pietro Berretta, Anza Lema, William Mbuta, Beni Tweve, Luca Brasili, Chiara Sinigaglia. Napak District: Gunther Nahrich, Apuda Daniel, Damiano Amei. Oyam District: Mattia Quargnolo, Samuel Okori, Christopher Bingom, Bobbie Okello J., Annet Ariko, Babra Muga. We also would like to thank Mauro Almaviva for his useful comments and insights.

Citation: Tavoschi L, Belardi P, Mazzilli S, Manenti F, Pellizzer G, Abebe D, et al. (2022) An integrated hospital-district performance evaluation for communicable diseases in low-and middle-income countries: Evidence from a pilot in three sub-Saharan countries. PLoS ONE 17(3): e0266225. https://doi.org/10.1371/journal.pone.0266225

References

1. Abbafati C, Machado DB, Cislaghi B, Salman OM, Karanikolos M, McKee M, et al. Global burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. 2020;396: 1204–1222. pmid:33069326

2. World Health Organization. Accelerating progress on HIV, tuberculosis, malaria, hepatitis and neglected tropical diseases. 2015.

3. Floyd K, Fitzpatrick C, Pantoja A, Raviglione M. Domestic and donor financing for tuberculosis care and control in low-income and middle-income countries: An analysis of trends, 2002–11, and requirements to meet 2015 targets. Lancet Glob Heal. 2013;1: e105–e115.

4. Resch S, Ryckman T, Hecht R. Funding AIDS programmes in the era of shared responsibility: An analysis of domestic spending in 12 low-income and middle-income countries. Lancet Glob Heal. 2015;3: e52–e61. pmid:25539970

5. Murray CJL, Evans D. Health systems performance assessment: goals, framework and overview. Debates, Methods and Empiricism. World Health Organization. World Health Organization; 2003.

6. Smith P, Mossialos E, Papanicolas I, Leatherman S. Performance measurement and professional improvement. Cambridge University Press. Cambridge University Press, Cambridge, UK; 2009. https://doi.org/10.1017/CBO9780511711800.022

7. World Health Organization. The World Health Report 2000. Health Systems: Improving Performance. World Heal Organ Rep. 2000.

8. Arah OA, Klazinga NS, Delnoij DMJ, Ten Asbroek AHA, Custers T. Conceptual frameworks for health systems performance: A quest for effectiveness, quality, and improvement. Int J Qual Heal Care. 2003;15: 377–398. pmid:14527982

9. Nuti S, Noto G, Vola F, Vainieri M. Let’s play the patients music: A new generation of performance measurement systems in healthcare. Manag Decis. 2018;56: 2252–2272.

10. Gray M, Pitini E, Kelley T, Bacon N. Managing population healthcare. J R Soc Med. 2017;110: 434–439. pmid:29148874

11. Bhattacharyya O, Mossman K, Ginther J, Hayden L, Sohal R, Cha J, et al. Assessing health program performance in low- and middle-income countries: Building a feasible, credible, and comprehensive framework. Global Health. 2015;11. pmid:26690660

12. Veillard J, Cowling K, Bitton A, Ratcliffe H, Kimball M, Barkley S, et al. Better Measurement for Performance Improvement in Low- and Middle-Income Countries: The Primary Health Care Performance Initiative (PHCPI) Experience of Conceptual Framework Development and Indicator Selection. Milbank Q. 2017;95: 836–883. pmid:29226448

13. Kruk ME, Gage AD, Arsenault C, Jordan K, Leslie HH, Roder-DeWan S, et al. High-quality health systems in the Sustainable Development Goals era: time for a revolution. Lancet Glob Heal. 2018;6: e1196–e1252. pmid:30196093

14. Suthar AB, Khalifa A, Joos O, Manders EJ, Abdul-Quader A, Amoyaw F, et al. National health information systems for achieving the Sustainable Development Goals. BMJ Open. 2019;9: 1–14.

15. Mbondji PE, Kebede D, Soumbey-Alley EW, Zielinski C, Kouvividila W, Lusamba-Dikassa PS. Health information systems in Africa: Descriptive analysis of data sources, information products and health statistics. J R Soc Med. 2014;107: 34–45. pmid:24914127

16. World Health Organization. Global tuberculosis report. 2020.

17. World Health Organization. Progress report on HIV, viral hepatitis and sexually transmitted infections. World Heal Organ. 2019.

18. World Health Organization. Xpert MTB/RIF implementation manual. Technical and operational ‘how-to’: practical considerations. World Heal Organ. 2004.

19. World Health Organization. Key TB indicators for individual countries and territories, WHO regions and the world. World Heal Organ. 2015.

20. World Health Organization. Consolidated strategic information guidelines for HIV in the health sector. 2015.

21. World Health Organization. The use of antiretroviral drugs for treating and preventing HIV infection. World Heal Organ. 2016.

22. World Health Organization. Infection Prevention and Control Assessment Framework At the Facility Level. 2018.

23. World Health Organization, UNICEF. Core questions and indicators for monitoring WAUnited Nations Children’s Fund World Health OrganizationSH in health care facilities in the Sustainable Development Goals. 2018.

24. Veillard J, Champagne F, Klazinga N, Kazandjian V, Arah OA, Guisset AL. A performance assessment framework for hospitals: The WHO regional office for Europe PATH project. Int J Qual Heal Care. 2005;17: 487–496. pmid:16155049

25. Nuti S, Vola F, Bonini A, Vainieri M. Making governance work in the health care sector: Evidence from a “natural experiment” in Italy. Heal Econ Policy Law. 2016;11: 17–38. pmid:25819303

26. Goodman DC, Morden NE, Ralston SL, Chang C-H, Parker DM, Weinstein SJ. The Dartmouth Atlas of Children’s Health Care in Northern New England. 2013.

27. OECD. Health at a Glance 2015. OECD Publ Paris. 2015. http://dx.doi.org/10.1787/health_glance-2015-en

28. OECD/European Observatory on Health Systems and Policies. United Kingdom: Country Health Profile 2019, State of Health in the EU. OECD Publishing, Paris/European Observatory on Health Systems and Policies, Brussels. OECD and World Health Organization; 2019.

29. Uplekar M, Weil D, Lonnroth K, Jaramillo E, Lienhardt C, Dias HM, et al. WHO’s new End TB Strategy. Lancet. 2015;385: 1799–1801. pmid:25814376

30. World Health Organization. Global Health Sector Strategy on HIV 2016–2021. Towards ending AIDS. World Heal Organ. 2016.

31. World Health Organization. WHO global strategy on people-centred and integrated health services: interim report. 2015.

32. Nuti S, De Rosis S, Bonciani M, Murante AM. Rethinking healthcare performance evaluation systems towards the people-centredness approach: Their pathways, their experience, their evaluation. Healthc Pap. 2017;17: 56–64. pmid:29595446

33. Fox MP, Rosen S. Patient retention in antiretroviral therapy programs up to three years on treatment in sub-Saharan Africa, 2007–2009: Systematic review. Trop Med Int Heal. 2010;15: 1–15. pmid:20586956

34. World Health Organization. Consolidated guidelines onthe use of antiretroviral drugs for treating and preventing hiv infection. Recommendations for a public health approach. 2016.

35. Chard AN, Gacic-Dobo M, Diallo MS, Samir, Sodha V, Wallace AS. Routine vaccination coverage—Worldwide, 2019. 2020. Report No.: 69(45), 1706.

36. Jha P. Reliable direct measurement of causes of death in low-and middle-income countries. BMC Med. 2014;12: 1–10. pmid:24495839

37. UNAIDS. Seizing the moment: Tackling entrenched inequalities to end epidemics. 2020.

38. UNAIDS. 90-90-90: An ambitious treatment target to help end the AIDS epidemic. 2020.

39. Noto G, Corazza I, Kļaviņa K, Lepiksone J, Nuti S. Health system performance assessment in small countries: The case study of Latvia. Int J Health Plann Manage. 2019;34: 1408–1422. pmid:31090962

40. Li H, Barsanti S, Bonini A. Building China’s municipal healthcare performance evaluation system: A Tuscan perspective. Int J Qual Heal Care. 2012;24: 403–410. pmid:22687705

41. Westert GP, Groenewoud S, Wennberg JE, Gerard C, Dasilva P, Atsma F, et al. Medical practice variation: Public reporting a first necessary step to spark change. Int J Qual Heal Care. 2018;30: 731–735. pmid:29718369

42. Fanelli S, Lanza G, Enna C, Zangrandi A. Managerial competences in public organisations: The healthcare professionals’ perspective. BMC Health Serv Res. 2020;20: 1–9. pmid:32293450

43. Wild L, King M, Chambers V, Harris D. Common constraints and incentive problems in service delivery. Overseas Dev Inst Work Pap. London; 2012.