Protocol for an evaluation of the initiation of an integrated longitudinal outpatient care model for severe chronic non-communicable diseases (PEN-Plus) at secondary care facilities (district hospitals) in 10 lower-income countries

Abstract

Introduction

The Package of Essential Noncommunicable Disease Interventions—Plus (PEN-Plus) is a strategy decentralising care for severe non-communicable diseases (NCDs) including type 1 diabetes, rheumatic heart disease and sickle cell disease, to increase access to care. In the PEN-Plus model, mid-level clinicians in intermediary facilities in low and lower middle income countries are trained to provide integrated care for conditions where services traditionally were only available at tertiary referral facilities. For the upcoming phase of activities, 18 first-level hospitals in 9 countries and 1 state in India were selected for PEN-Plus expansion and will treat a variety of severe NCDs. Over 3 years, the countries and state are expected to: (1) establish PEN-Plus clinics in one or two district hospitals, (2) support these clinics to mature into training sites in preparation for national or state-level scale-up, and (3) work with the national or state-level stakeholders to describe, measure and advocate for PEN-Plus to support development of a national operational plan for scale-up.

Methods and analysis

Guided by Proctor outcomes for implementation research, we are conducting a mixed-method evaluation consisting of 10 components to understand outcomes in clinical implementation, training and policy development. Data will be collected through a mix of quantitative surveys, routine reporting, routine clinical data and qualitative interviews.

Ethics and dissemination

This protocol has been considered exempt or covered by central and local institutional review boards. Findings will be disseminated throughout the project’s course, including through quarterly M&E discussions, semiannual formative assessments, dashboard mapping of progress, quarterly newsletters, regular feedback loops with national stakeholders and publication in peer-reviewed journals.

AJA and EBW are joint first authors.

GB and AOM are joint senior authors.

STRENGTHS AND LIMITATIONS OF THIS STUDY

• This is a large prospective study carried out across 18 sites in 10 countries.

• It has a comprehensive mixed-method design.

• The main limitation is that it is an observational study with no control sites.

Introduction

Background

PEN-Plus (the Package of Essential Noncommunicable Disease Interventions—Plus) is a strategy to increase access to services and save lives by decentralising care for severe chronic non-communicable diseases (NCDs) in highly constrained health systems.^{1 2} These severe chronic NCDs (SC-NCDs) include conditions such as type 1 diabetes (T1D), rheumatic heart disease and sickle cell disease (SCD) that require services such as echocardiography, management of insulin and prescription of heart failure medications, anticoagulation and hydroxyurea that are rarely available at lower level health facilities in lower-income countries. PEN-Plus enables mid-level providers, such as nurses and clinical officers, to provide integrated care for a set of related conditions at intermediate care facilities such as district hospitals. PEN-Plus builds on and strengthens the WHO PEN strategy for decentralisation of care for common NCDs, such as hypertension, asthma and type 2 diabetes, at primary healthcare facilities.³

PEN-Plus was initially developed and scaled nationally in Rwanda.^{4 5} PEN-Plus strategies have since been initiated in Haiti, Malawi and Liberia. Research has shown PEN-Plus to be affordable,⁴ effective,⁶ acceptable^{7 8} and scalable,⁵ but questions still remain about how to implement in new settings and how to scale up. PEN-Plus was identified by the Lancet Commission on Reframing NCDs and Injuries for the Poorest Billion (NCDI Poverty Commission) as a promising example of care through integrated teams. In 2019, WHO held a consultation to review a draft PEN-Plus strategy for the African Region.³ In 2022, the WHO Regional Office of Africa adopted the PEN-Plus strategy as part of its Seventy-second Regional Committee Meeting.⁹ In late 2020, the 21-country NCDI Poverty Network was launched to support implementation of the Lancet Commission’s recommendation. The Center for Integration Science at Brigham and Women’s Hospital serves as the Network’s co-secretariat together with Mozambique’s Universidade Eduardo Mondale. Increasing implementation of PEN-Plus is one of Network’s strategic initiatives. Now, nine new countries and one state in India are initiating PEN-Plus programmes with support from the Network and other partners.

PEN-Plus expansion

A request for applications inviting countries to submit a letter of interest to apply for PEN-Plus implementation was posted and shared among partners in July 2020. Implementation organisations from countries that had previously completed a National NCDI Poverty Commission in partnership with the NCDI Poverty Network were eligible to apply. We received 21 applications from organisations in 11 countries. Following in-depth discussions and strategic planning from June to August 2021, 10 implementing partners were chosen to develop full proposals detailing implementation of a PEN-Plus package of clinical services at first-level hospitals.

Eighteen first-level hospitals in nine countries and one Indian state were selected for PEN-Plus expansion (table 1). The participating hospitals differ in the organisation of service delivery, training and overall systems in place. Based on these differing systems, we expect to have a range of approaches for different components of the implementation. In addition, we have a differing baseline knowledge of what existing services are. Where available, pre-existing services for NCDs are largely limited to primary care including screening and management of more common, less complex NCDs such as hypertension and type 2 diabetes, with little treatment for SC-NCDS that are the target of this work. Interventions contained in PEN-Plus vary based on local priorities (table 2), and existing Ministry of Health (MoH) initiatives. All PEN-Plus programmes involve training of midlevel providers and ensuring access to appropriate devices and medications for appropriate diagnosis and treatment of SC-NCDs.²

Table 1

PEN-Plus clinical sites

Country	Implementing partner	PEN-Plus clinical site	Facility type	Catchment population
Chhattisgarh State, India	Chhattisgarh NCD Plus Initiative	Surguja Government District Hospital, Surguja District (located at Ambikapur)	Public	840 352
		Surajpur Government District Hospital, Surajpur District	Public	900 198
Ethiopia	Mathiwos Wondu—Ye Ethiopia Cancer Society	Muketuri Primary Hospital, Oromia Region	Public	304 749
		Addis Zemen Primary Hospital, Amhara Region	Public	500 000
Kenya	NCD Alliance Kenya (NCDAK)	Hamisi Subcounty Hospital, Vihiga County	Public	176 264
		Kinna Health Center, Isiolo County	Public	115 533
Mozambique	Universidade Eduardo Mondlane (UEM) + Instituto Nacional de Saúde (INS) + CUAMM + Mozambique Institute for Health Education and Research (MIHER)	Hospital Rural de Nhamatanda, Nhamatanda District	Public	317 538
Nepal	Kathmandu Institute of Child Health (KIOCH)	Bardiya District Hospital, Bardiya District, Lumbini Province	Public	426 576
		Damak Hospital, Jhapa District, Province No. 1	Public	300 000
Sierra Leone	Partners In Health	Koidu Government Hospital, Kono District	Public	620 703
	CUAMM	Pujehun Hospital, Pujehun District	Public	406 931
Tanzania	National Institute for Medical Research (NIMR)	Kondoa District Hospital, Central Zone	Public	260 000
Uganda	Uganda Initiative for Integrated Management of Non-Communicable Diseases (UINCD)	Nakaseke General Hospital, Nakaseke District, Central Region	Public	300 000
		Atutur General Hospital, Kumi District, Eastern Region	Public	301 200
Zambia	Centre for Infectious Diseases Research in Zambia (CIDRZ)	Chibombo District Hospital (Mwachisompola), Chibombo District, Central Province	Public	421 315
		Matero Level 1 Hospital, Lusaka District	Public	478 710
Zimbabwe	SolidarMed+Clinton Health Access Initiative (CHAI)	Ndanga District Hospital, Zaka District	Public	194 739
		Mashoko Christian Hospital, Bikita District	Faith-based	174 068

PEN-Plus, Package of Essential Noncommunicable Disease Interventions—Plus.

Table 2

Evaluation components

Evaluation component	Research question	Timeline	Data collection lead	Proctor outcomes
Monitoring and evaluation	1,2	Quarterly	Site and Central	Implementation (adoption, feasibility, fidelity, penetration), service outcomes, client outcomes
Training indicators	2	Quarterly	Site and Central	Implementation (adoption, feasibility, fidelity, penetration)
Retrospective record reviews	1	2023, 2024	Site	Service outcomes (effectiveness), client outcomes
Baseline assessment	1	Q1 2022	Site	Implementation (adoption, fidelity, penetration), client outcomes
Qualitative formative assessments	1, 2, 3	Semiannual	Central	Implementation (acceptability, appropriateness, feasibility)
Midline assessment	1,2	2023	Site	Implementation (adoption, fidelity, penetration)
Endline assessment	1,2	2024–2025	Site	Implementation (adoption, fidelity, penetration),
Policy maker interviews	3	2022, 2024	Central	Implementation (acceptability, appropriateness, sustainability), health system
Cost analysis	1	Ongoing	Site and Central	Implementation (cost)
Monitoring of MoH activities	3	Ongoing	Central	Health system

Over 3 years, these sites are expected to conduct the following activities:

• Establish PEN-Plus clinics in one or two district hospitals, at least one of which must be rural. Clinics are expected to be staffed with trained staff and at a minimum provide chronic care for patients with severe or complex NCDs including type 1 and insulin-dependent diabetes, rheumatic and congenital heart disease and SCD. Based on sites-specific priorities, burden and other national resources, other conditions may be included as shown in online supplemental appendix 2. PEN-Plus clinics establishment may entail:

  • Hiring, supervising and supporting clinical and auxiliary staff.

  • Staff training and mentorship.

  • Supplies and commodities procurement and distribution, with any needed supply chain support (eg, helping with projections, reporting on supplies).

  • Supporting infrastructure where needed for clinic space.

  • Systems development and support (eg, M&E).

• Support the PEN-Plus clinics to mature into PEN-Plus training sites, in preparation for national or state-level scale-up. This would consist of readiness to host trainees that will be working to establish PEN-Plus clinics in other parts of the country.

• Work with national MoH and other national stakeholders to describe, measure and advocate for PEN-Plus, ultimately supporting the development of a national PEN-Plus operational plan for scale-up following the grant period. This work includes assessing gaps in relevant clinical guidelines, formularies, clinical forms, regulations around scope of practice for mid-level providers such as nurses and clinical officers, and adaptation of training materials for national use in support of the MoH.

Implementation activities

Training

Training models for PEN-Plus will rely on many in-country contextual factors such as existing training programmes, specialist and other trainer availability, geography and certification processes. Thus, different models of PEN-Plus training, adjusted for context, are anticipated. For foundational PEN-Plus training, each implementing partner is planning on a set of didactics or classroom learning targeting the conditions in the initial proposed PEN-Plus package of care. Some countries may use e-learning or other digital innovations to support the didactic component of training. To complement classroom learning, each country will also incorporate practical or hands-on models of training where PEN-Plus trainees are clinically supervised by physician specialists or other expert cadres as they see patients, until trainees are deemed ready to see patients at PEN-Plus clinics independently. For example, in the most optimal setting, a country may already have specialists, who would come in weekly or biweekly to provide clinical supervision during PEN-Plus clinical encounters. However, this will not be the case for each type of condition or specialist in each setting, so a variety of approaches will be needed such as clinical experts travelling to be on-site for larger blocks of time, either from urban centres in-country or internationally.

We anticipate that most countries will have a hybrid combination of these models, as well as the potential for remote supervision via telemedicine and that systems will mature and advance over time. A large deliverable of the joint work across the 10 countries/states will involve articulating these different models, resources needed and best practices to ensure PEN-Plus trainees complete foundational training with competencies and skills needed to care for patients independently.

Trainees will initially be trained on basic competencies including an understanding of the pathophysiology of PEN-Plus conditions, knowledge of symptoms and diagnostic criteria, protocols for treatment, and counselling and education. Intermediate competencies will include knowledge of local and international epidemiology, atypical presentations of the disease, and, where relevant, new or advanced treatment techniques or updated protocols. Trainees will be evaluated using pre-training and post-training evaluations on condition-specific skills including screening, testing and treatment of PEN-Plus conditions like HBA1C testing, diabetes self-management education, echocardiology testing and peak flow spirometry.

Clinical implementation, community engagement and case finding

The PEN-Plus expansion work offers tremendous opportunity for learning across geographic contexts and clinical settings. Teams will tackle implementation questions such as staff supervision and mentorship, procedures for how patients flow through clinic, patient screening and linkage to care from a variety of clinical settings, long-term follow-up and missed visit tracking systems, incorporation of digital tools, research questions and more. The project will create several platforms for sharing of lessons, challenges and best practices to maximise cross-site learning including technical learning sessions, data review meetings, on-site visits and more.

Community engagement will be variable between sites, but examples may include engagement with community leaders, training of community health workers, social workers, radio campaigns, school campaigns and outreach.

For case finding, clinics initially will focus on identifying patients coming in for acute care (which primarily involves building links with inpatient care teams, emergency rooms and acute care outpatient clinics) and training providers to understand the initial signs of PEN-Plus conditions. Clinics are also strengthening links with referral hospitals to enable patients being seen at referral hospitals for conditions including T1D or SCD to be counter-referred to PEN-Plus clinics closer to their home. In later phases, after development of systems to identify the acutely ill patients, clinics will start to explore community case finding. This case finding varies in each setting and for each condition, but examples include school education around signs of T1D, community health worker education and radio outreach programmes.

Procurement

The PEN-Plus clinics will need a larger set of equipment available than is likely to be routinely found at first-level hospitals in these settings. This includes but is not limited to imaging capacity such as ultrasound machines with probes for echocardiography and other types of ultrasound scanning and laboratory capacity such as haemoglobin A1C, electrolytes, kidney and liver function, anticoagulation monitoring and sickle cell diagnostic tests. Most included sites are public MoH affiliated hospitals. Training, medications and equipment across the 10 countries are being externally financed through grants and donations. Implementation is occurring in partnership with local and central Ministries of Health to identify needs, gaps and public systems so that in the longer-term systems will be incorporated into national systems. Many of the countries will be able to acquire equipment in country or through international orders, and others will be supported for the initial PEN-Plus clinics by central procurement functions. Equipment models, pricing, procurement pathways, maintenance plans and ongoing planning for reagents and consumables will be closely tracked as part of the collective learning from PEN-Plus expansion to inform future scale-up and additional expansion and financing initiatives.

National planning

Each team will work through the MoH and relevant government officials to support the development of a national PEN-Plus operational plan. Building on the work of the national NCDI Poverty Commissions in each setting,¹⁰ these operational plans will bring together key stakeholders to map out clear pathways for PEN-Plus expansion to first-level hospitals for national scale-up in the 10 years following initial pilot implementation of PEN-Plus. Each country will engage stakeholders across many realms including government, academic institutions, civil society and patient advocacy groups, clinical specialists, frontline healthcare workers, implementing partners and global and multilateral organisations. This process has already been conducted in Malawi, resulting in a national operational plan for PEN-Plus^11–13 as well as a national Steering Committee to steward and oversee national expansion.

Monitoring and evaluation

This project involves gathering patient-level and visit-level data for each of the PEN-Plus conditions. Preliminary information suggest that most or all PEN-Plus clinics have paper-based data collection systems and will gradually progress to electronic medical records when sites are ready. M&E activities for the 10 sites were designed such that all 10 sites could complete them, regardless of their data system maturity. Specific activities will include:

• Collection of Quarterly Core Indicators and Narrative—a template with PEN-Plus core indicators and a quarterly narrative section will be completed by implementing partners each quarter (online supplemental appendix 1). PEN-Plus core indicators focus on basic patient counts (eg, numbers enrolled, active, loss to follow-up, and deaths). After being audited, these core indicators will feed into a cross-site dashboard. The quarterly narrative will prompt the clinical team to reflect on challenges and successes during the previous quarter.

• M&E based discussions/interactions—Central M&E staff will facilitate a quarterly discussion during the monthly PEN-Plus partners meeting. In order to promote shared learning in PEN-Plus, local M&E staff persons will also be invited to share interesting quality improvement cycles, challenges and other materials.

Additionally, a data dictionary has been developed to promote the standardised collection of clinical data across the 18 sites to the extent possible. To aid in the standardised collection of clinical data, a comprehensive data dictionary will be shared with clinical sites to help guide the development of clinical forms at PEN-Plus clinics.

As WHO-AFRO has endorsed PEN-Plus throughout the African region, PEN-Plus is expected to launch in many settings. As PEN-Plus is being expanded in a relatively short time period to 18 sites across 10 countries, it is imperative to understand implementation challenges and regional differences in the acceptability, adoption, implementation and sustainability of PEN-Plus. These lessons will aid us in both ensuring best possible care at existing PEN-Plus sites, and ensuring optimal design for further expansion. This is a protocol describing the 10-component formative evaluation of the PEN-Plus initial programme of expansion into 10 new countries. The project will document and understand operational challenges to ensure best possible implementation at current implementation sites and inform further scale-up.

Methods and analysis

Research questions

This protocol is developed for a mixed methods evaluation based on the United Kingdom’s Medical Research Council (MRC) framework for complex interventions¹⁴ guided by the Proctor implementation framework.¹⁵ The evaluation is designed to reflect overarching research questions across all sites, but also allows for the considerable variability in implementation practices and needs across the nine countries and one state. Here, we aim to understand:

• Clinical implementation outcomes

  • Are PEN-Plus clinics enrolling sufficient numbers of patients, are these clinics concentrating on patients with severe chronic NCDs, particularly T1D, and are they reaching the poorest? (Sufficient numbers for each condition and clinic will be determined based on local prevalence in the catchment area, and other factors related to clinic capacity and accessibility).

  • Is the care being provided at these new clinics acceptable and comparable in quality to tertiary chronic care facilities in their country?

  • Are patients enrolled in these clinics being protected from further impoverishment?

  • What is the cost of providing these services?

• Training outcomes

  • Over the course of the project, what is the readiness of the supported facilities to become national training centres for PEN-Plus?

  • How many midlevel providers have been prepared to become PEN-Plus master trainers including a sufficient period of clinical practice precepted by internal medicine specialists, paediatricians, cardiologists and endocrinologists?

  • Is the PEN-Plus clinic space adequate for a training facility?

  • What is the 6-month retention of these midlevel providers?

• Policy outcomes

  • Has the MoH, together with its partners, begun to develop the policies and strategies needed to enable the implementation of PEN-Plus services at a national scale?

  • These policies and strategies include: disease management guidelines, medication formularies, patient medical record forms, health management information systems, training curricula and certification pathways for providers, national PEN-Plus operational plans and resource mobilisation plans.

Study design

This is a mixed-method evaluation consisting of 10 components (table 2). Data will be collected through a mix of quantitative surveys (baseline, midline and endline surveys), routine reporting (M&E and training indicators), routine clinical data (retrospective record reviews) and qualitative interviews.

Study area and study site selection

This evaluation will occur in the 18 sites in ten countries where PEN-Plus expansion is taking place: Chhattisgarh State, India, Ethiopia, Kenya, Mozambique, Nepal, Sierra Leone, Tanzania, Ugana, Zambia and Zimbabwe (table 1). First-level hospitals were selected by the MoH and implementation partners in each country. We provided guidance including a focus on the rural poor, and a catchment population typically between 100 000 and 50 000 people. Using those criteria, the MoH and partners chose hospitals based on contextualised factors including local MoH priorities, local political will and commitment, epidemiology and geography. These factors are expected to be typical and define challenges for PEN-Plus implantation across the region and into the future.

Study period

This research will be carried out over a 3-year period between 2022 and 2025 (table 2).

Sampling

No sampling will be conducted as all clinics initiating PEN-Plus during the 3-year period and all patients with PEN-Plus conditions will be included.

Evaluation framework

We will use the Proctor Implementation framework and Institute of Medicine (IOM) standards of care¹⁶ to understand the implementation outcomes including acceptability, adoption, appropriateness, feasibility, fidelity, penetration and sustainability. In addition to the implementation outcomes, we will also assess changes to service and client outcomes, as noted below.

Theory of change

Following MRC guidance for the evaluation of complex interventions, we adapted the theory of change based on Proctor characteristics (figure 1).¹⁵

Figure 1. Theory of change. NCD, non-communicable diseases; PEN-Plus, Package of Essential Noncommunicable Disease Interventions—Plus.

Evaluation components

Data collection

We will conduct both routine data collection and new data collection activities. We briefly describe both below (table 2, online supplemental appendix 3) Data collection will be done both centrally and at the site level, using paper-based forms, RedCap and excel spreadsheets.

Routine data collection

Routine monitoring and evaluation

Routine data collection will begin when the clinics are running. The M&E section of the PEN-Plus implementation involves quarterly review and reflection on collected information to inform formative strategic, evidence-based improvements to the implementation efforts. As site M&E teams submit collected data on indicators, the data will be checked for missingness, outliers and errors. The team will communicate with individual site teams to resolve questions related to data quality and completeness. Additionally, the team will meet with each site quarterly to discuss trends and identify and troubleshoot issues related to data collection. These calls will also provide a space for implementers to share their on-the-ground experiences and reflections, and potential quality improvement cycles.

Indicators will include (by condition and health facility):

• Total patients ever enrolled.

• Total patients currently enrolled and active in care.

• Total patients <18 years old.

• Total patients who are female.

• Active patients who had a visit in last quarter.

• New patients enrolled in last quarter.

• Patients who have been lost to follow-up in last quarter.

• Patient deaths reported in last quarter.

Training

Training indicators will be collected quarterly with the goal of understanding the number of providers (by cadre) capable of providing PEN-Plus services. We will have a core set of quarterly indicators and will add other indicators as the systems mature. We will gather information including:

• Midlevel providers fully trained for PEN-Plus as per local definitions.

• Midlevel providers fully trained and currently working in PEN Plus clinic(s).

• Midlevel providers currently enrolled in classroom or didactic training.

• Midlevel providers currently enrolled in e-learning course.

• Midlevel providers currently enrolled in in-person didactic course.

• Mentorship visit days this quarter to the PEN-Plus clinic(s) by specialists (total).

• Trained providers working in PEN-Plus clinics 6 months post training.

Depending on the unique reporting structure of the individual sites, the person reporting may be a member of the training team, the in-person specialist, and/or the master trainer coordinating training.

Retrospective record reviews

We will document the site-specific patient cohorts using retrospective record reviews, or electronic record analysis (depending on site capability). Anticipating a majority of sites will begin with paper-based records; a retrospective review will be performed once during implementation and once at the end of the 3-year period. Retrospective record reviews will be conducted at the site level through manually inspecting routine patient records and collecting information on patient outcomes for key conditions.

Data from the patient cohort will include:

• Number of patients being treated.

• Costs to the patient.

• Availability of PEN-Plus to the poorest and sickest patients.

• Patient progress.

• Patient-centeredness.

• Clinical characteristics and outcomes.

New data collection

Baseline assessments

The baseline assessments is designed to understand the overall structure and function of the hospitals that will be part of PEN-Plus expansion. The baseline assessment was designed centrally with site base input, and consists of questions about hospital structure, staffing, infrastructure, data systems and the availability of medicines, diagnostics and equipment. We expect to administer the baseline survey in the first half of 2022. The baseline assessment is created in RedCap and sites will input data either directly into RedCap or using paper based forms. If paper base forms are used, then they will be enterered into RedCap manually centrally.

Qualitative formative assessments

Twice a year, the central team will conduct structured calls with members of the NCD team to understand any barriers and facilitators to PEN-Plus implementation. The calls will also provide an opportunity to work through issues and understand best practices to use for future scale-up. Topics will be kept open to enable us to explore early experiences, but are expected to cover perceived challenges, what has worked well, what has worked less well and any other issues that stakeholders have not had other opportunities to address.

Midline surveys

One year into the implementation, we will conduct an interim midline assessment that will consist of readministering a simplified version of the baseline survey to understand improvements to clinic structure, staffing, supply chain and infrastructure. Data collected at this time will be used to identify bottlenecks and allow for timely course correction. Forms will be created in RedCap, and individuals at the site level will input data either directly into RedCap or into paper-based forms. If paper base forms are used, then they will be enterered into RedCap manually centrally.

Endline survey

At the end of the initation project, we will conduct an endline assessment consisting of readministering a simplified version of the baseline assessment to understand institutional progress made to implementing PEN-Plus, and progress to becoming a national training site. Forms will be created in RedCap, and individuals at the site level will input data either directly into RedCap or into paper-based forms. If paper base forms are used, then they will be enterered into RedCap manually centrally

Policy interviews

Qualitative policy interviews will be conducted centrally twice; at the beginning if implementation and at the end of the the 3-year cycle with MoH officials and other national stakeholders active in agenda setting in NCDs at the national level. Informants will vary by country reflecting the unique makeup of the national policy space. Interviews will be conducted at the beginning of the project and after 3 years of implementation to understand local ownership and how early experiences have affected readiness and acceptability of national expansion of PEN-Plus. Questions will assess familiarity with PEN-Plus, perceptions of success, opportunities, dedication to training and barriers to successful scale-up. Qualitative data will be collected through interviews and may be conducted in person or via an online video platform such as zoom. Interviews will be conducted in English, or, if necessary, with the use of a translator. With consent, interviews will be recorded and transcribed.

Cost analysis

We will review payroll, budgets and other financial documentation available at each facility. Based on this information and to the extent possible, we will estimate the total annualised cost of services across all patients by summating estimated costs for several cost categories—including personnel, equipment, infrastructure and consumables. Coupled with outpatient and inpatient service volume, we will then generate an average cost per patient per visit for outpatient services and an average cost per patient per diem for inpatient services.

Outcomes

Our research questions span clinical and national levels. At the clinic level we are interested in (1) improvements in clinical implementation outcomes, including correct diagnosis, initiation, and continuation of quality treatment for SC-NCDs; and (2) readiness and progress in training outcomes, including progress towards being national training centres. At the national level, we are interested in (3) policy outcomes, including sustainability and appetite for PEN-Plus scale-up at the national level.

For the purposes of this evaluation, we categorise our indicators into implementation, service and patient outcomes. Due to considerable heterogeneity across sites, progress will be measured differently based on local site characteristics. Outcomes mapped to the research question they pertain to and the evaluation component they are part of are found in table 3.

Table 3

Outcomes to research questions

Question	Implementation outcome(s)	Service outcome(s)	Client outcome(s)	Evaluation component
Clinical Implementation Outcomes
Are PEN-Plus clinics enrolling sufficient numbers of patients largely with severe chronic non-communicable diseases, and particularly type 1 diabetes?	Fidelity Penetration	Effectiveness Equity		Retrospective record reviews
Is the care being provided at these new clinics acceptable and comparable in quality to tertiary chronic care facilities in their country?		Safety Effectiveness	Function Symptomatology Satisfaction	Formative assessments, M&E, published reports
Are patients enrolled in these clinics being protected from further impoverishment?		Equity	Access Financial burden	Retrospective record reviews
What is the cost of providing these services?	Costs			Cost analysis
Are patients’ voices being heard?		Patient centeredness		Retrospective record reviews
Training outcomes
What is the readiness of the supported facilities to become national training centres for PEN-Plus?	Adoption Appropriateness Feasibility Fidelity Penetration Sustainability			Baseline, M&E, Training, formative assessments, midline, endline, policy interviews
How many midlevel providers have been prepared to become PEN-Plus master trainers including a sufficient period of clinical practice precepted by internal medicine specialists, paediatricians, cardiologists and endocrinologists	Adoption Fidelity Penetration			M&E, Training
Is the PEN-Plus clinic space adequate for a training facility	Appropriateness Feasibility			Baseline, Midline, Endline
What is the 6-month retention of these midlevel providers?	Sustainability			Training
Policy outcomes
Has the Ministry of Health (MoH), together with its partners, developed the policies and strategies needed to enable implementation of PEN-Plus services at a national scale?	Acceptability Adoption Penetration Sustainability			Policy interviews, Monitoring of MoH activities

Study participants

The primary unit of analysis of this study will be the implementing first-level hospital, except for clinical outcomes where the patients will be the unit of analysis.

Data management

Data will be collected through paper-based forms or Research Electronic Data Capture (REDCap) electronic data capture tools hosted at Mass General Brigham.^{17 18} All data will be deidentified prior to data transfer. Data will be shared either through secure email or institutional Dropbox.¹⁹ Data will be stored on secure Dropbox, and pooled non-identifiable indicator data will be uploaded into a shared cross-site dashboard powered by Microsoft Power BI.²⁰ All computers used to access data will be secure and password protected. Any paper data collection forms will be held in locked filing cabinets and will be stored for a maximum of 7 years, or end of research activities. All forms will be securely destroyed at the end of this period.

Data analysis

Baseline, Midline and Endline Assessments will be collected in RedCap. We will use excel to analyse descriptive statistics in order to understand trends and the evolution of clinic space, staffing, medicine and equipment availability and the clinics adequacy to become a training facility

Cost data will be collected on excel spreadsheets and simple descriptive statistics will be used to understand broad costs.

M&E and training data will be collected in Excel based sheets and used to enable timely reporting and understanding sites progress towards implementing PEN-Plus and ability to become regional training centres. We will examine trends over time and progress towards site-specific targets.

Retrospective record reviews will include patient numbers trends over time, and progress towards desirable patient outcomes for key conditions. Data will be collected in excel and analysed using descriptive statistics,

Qualitative data

After obtaining informed consent, interviews will be conducted virtually and, with permission, recorded. Qualitative data will be analysed in Dedoose²¹ using a thematic analysis. Themes will be identified a priori and after initial coding based on these themes, we will expand our coding structure to encompass additional themes that emerged through the first step of analysis.

Data checking

Data will be checked for consistency and anomalous findings will be flagged and discussed with the local M&E or training teams.

Patient and public involvement

PEN-Plus is a programme aimed at improving care and the lives of people living with SC-NCDs, and was developed with the assistance of People living with SC-NCDs. Many of the authors and authorship group are either PLWSCNCDs or advocates. Patient advocates will be involved in dissemention activities.

Ethics and dissemination

This protocol has been considered exempt or covered by Mass General Brigham #2022P001390, Nepal Health Research Council 571/2022 P, National Institute for Medical Resarch (Tanzania) NIMR/Hq/R.8aVol.IX/4184, University of Zambia Biomedical Research Ethics Committee #3032-2022, Mulago Hospital Research and Ethics Committee MHREC 2022-74, Amhara Public Health Institute NoH/R/T/T/D/07/43 and Oromia Health Care Bureau BF/HBTFH/H6/2027.

Findings from routine data collection will be fed back to the local sites during quarterly M&E discussions and semiannual formative assessments. Progress will be mapped on a dashboard that sites can access at any time. We will send quarterly newsletters to all the sites highlighting statistics, progress or successes including characteristics and predictors of compliance. We will have regular feedback loops with national stakeholders including members of the MoH. Finally, we will disseminate our findings and best practice through publications in open-access peer-reviewed journals.

Ethics statements

Patient consent for publication

Not applicable.

Footnote

Twitter @reubenmutagaywa

Collaborators The PEN-Plus Partnership study group includes key stakeholders and contributing researchers from 10 countries initiating PEN-Plus strategy. These individuals contributed to all sections of this research and are listed by country: Abhijit Gadewar, Sunil Jadhao, Yogeshwar Kalkonde, Chetanya Malik (Chhattisgarh, India); Alma Adler, Victoria Bhambhani, Chantelle Boudreaux, Lauren Brown, Gene Bukhman, Matthew M. Coates, Wubaye Walelgne Dagnaw, Laura Drown, Susan Donnellan, Gina Ferrari, Apoorva Gomber, Neil Gupta, Yogesh Jain, Kaita Domingues, Sheila Klassen, Gene Kwan, Andrew Marx, Ryan McBain, Amy McLaughlin, Emmanuel Mensah, Gedeon Ngoga, Maia Olsen, Catherine Player, Ramon Ruiz, Devashri Salvi, Ada Thapa, Celina Trujillo, Leslie Wentworth, Allison Westervelt, Ariana Wolgin, Emily B. Wroe, Emily Yale (Center for Integration Science, NCDI Poverty Network Co-Secretariat); Michael Abiyu, Lemma Ayele, Wondu Bekele, Zelalem Mengistu, Temesgen Sileshi, Natnael Alemayehu, Natnael A. Abebe (Ethiopia); Dawson Calixte, Darius Fenelon, Nancy Larco (Haiti); Catherine Karekezi, Gideon Ayodo, Gladwell Gathecha, Peter Mokaya, Lilian Mbau, Zipporah Ali (Kenya); Sterman Toussaint (Liberia); Jones Masiye, Todd Ruderman, Chiyembekezo Kachimanga, Evelyn Chibwe, Noel Kasomekera (Malawi); Ana Mocumbi, Emílio Tostão, Beatriz Manuel, Neusa Bay, Sam Patel, Humberto Muquingue, Sergio Chicumbe, Nicole M Salipa, Riaze Rafik, Lucy Ramirez, Andrea Atzori, Fabio Manenti, Giovanni Putoto (Mozambique); Shiva Adhikary, Krishna Aryal, Phanindra Baral, Meghnath Dhimal, Biraj Karmachaya, Bhagawan Koirala, Abha Shrestha, Archana Shrestha, Abhinav Vaidhya (Nepal); Symaque Dusabeyezu, Innocent Kamali (Rwanda); Remy Bitwayiki Nkwiro, Marta Patiño, Santigie Sesay (Sierra Leone); Mary Mayige, Reuben Mutagaywa, Julie Makani (Tanzania); Isaac Ssinabulya, Ann Akiteng, Robert Kalyesubula, Frank Mugabe, Sarah Asio Eragu, Bernard Bukar (Uganda); Roma Chilengi, Fastone Mathew Goma, Namasiku Siyumbwa, Wibroad Mutale (Zambia); Kudakwashe Madzeke, Alexio Mangwiro, Alaisa Mbiriri, Alvern Mutengerere, Wencelas, Nyamayaro, Porika Nyawai, Abaden Svisva, Laura Ruckstuhl, and Lucia Gonzalez (Zimbabwe). All authors approved the final manuscript and are solely responsible for its accuracy.

Contributors All authors shared an equal degree of contribution to the study conception and design. Equal contribution for draft manuscript preparation: AJA, EBW, GB and AOM. All authors reviewed and approved the final version of the manuscript. GB and AOM are joint last authors.

Funding This work was supported by The Leona M. and Harry B. Helmsley Charitable Trust grant number (2206-05274).

Competing interests None declared.

Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

References

¹ Bukhman G, Kidder A, eds. The Partners In Health guide to chronic care integration for endemic non-communicable diseases. Rwanda edition. Cardiac, renal, diabetes, pulmonary, and palliative care. Boston, MA: Partners In Health, 2011.

² Bukhman G, Mocumbi A, Wroe E, et al. The PEN-plus partnership: addressing severe chronic non-communicable diseases among the poorest billion. Lancet Diabetes Endocrinol 2023; 11: 384–6. doi:10.1016/S2213-8587(23)00118-3

³ World Health Organization Regional Office for Africa. Report on regional consultation: WHO PEN and integrated outpatient care for severe, chronic NCDS at first referral hospitals in the African region (PEN-plus). 2019. Available: https://www.afro.who.int/sites/default/files/2020-01/PENPEN-Plus%20Meeting%20Report%20-%20WHO%20AFRO%20Rwanda%20July%202019%20w%20updated%20Photos.pdf

⁴ Eberly LA, Rusangwa C, Ng’ang’a L, et al. Cost of integrated chronic care for severe non-communicable diseases at district hospitals in rural Rwanda. BMJ Glob Health 2019; 4: e001449. doi:10.1136/bmjgh-2019-001449

⁵ Niyonsenga SP, Park PH, Ngoga G, et al. Implementation outcomes of national decentralization of integrated outpatient services for severe non-communicable diseases to district hospitals in Rwanda. Trop Med Int Health 2021; 26: 953–61. doi:10.1111/tmi.13593

⁶ Ruderman T, Chibwe E, Boudreaux C, et al. Training mid-level providers to treat severe non-communicable diseases in Neno, Malawi through PEN-plus strategies. Ann Glob Health 2022; 88: 69. doi:10.5334/aogh.3750

⁷ Drown L, Adler AJ, Schwartz LN, et al. Living with type 1 diabetes in Neno, Malawi: a qualitative study of self-management and experiences in care. BMC Health Serv Res 2023; 23: 595. doi:10.1186/s12913-023-09519-z

⁸ Adler AJ, Trujillo C, Schwartz L, et al. Experience of living with type 1 diabetes in a low-income country: a qualitative study from Liberia. BMJ Open 2021; 11: e049738. doi:10.1136/bmjopen-2021-049738

⁹ Boudreaux C, Barango P, Adler A, et al. Addressing severe chronic NCDs across Africa: measuring demand for the package of essential non-communicable disease interventions-plus (PEN-plus). Health Policy Plan 2022; 37: 452–60. doi:10.1093/heapol/czab142

¹⁰ NCDI Poverty Network. Overview. Available: http://www.ncdipoverty.org/overview1 [Accessed 10 Jan 2022 ].

¹¹ Malawi PEN-plus operational plan. Lilongwe, Malawi Malawi Ministry of Health & Population; 2021. Available: https://static1.squarespace.com/static/55d4de6de4b011a1673a40a6/t/60d9f30fbd36ae7b031f346d/1624896278874/Malawi+PEN-Plus+Operational+Plan_FINAL_23-June-2021.pdf

¹² Marx A. Malawi launches operational plan for national scale-up. Boston (USA) NCDI Poverty Network; 2021. Available: http://www.ncdipoverty.org/blog/2021/6/28/malawi-launches-operational-plan-for-national-scale-up-of-pen-plus [Accessed 10 Jan 2022 ].

¹³ Mtawali W. Malawi launches WHO PEN-plus plan to reduce burden of NCD. Malawi Nyasa Times; 2021. Available: https://www.nyasatimes.com/malawi-launches-who-pen-plus-plan-to-reduce-burden-of-ncd/ [Accessed 10 Jan 2022 ].

¹⁴ Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions. Medical Research Council. Available: https://webarchive.nationalarchives.gov.uk/ukgwa/20220207162925/http://mrc.ukri.org/documents/pdf/complex-interventions-guidance/ [Accessed 10 Jan 2022 ].

¹⁵ Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011; 38: 65–76. doi:10.1007/s10488-010-0319-7

¹⁶ Wakefield MK. The quality chasm series: implications for nursing. In: Hughes RG, ed. Patient safety and quality: an evidence-based handbook for nurses, Table 2, [IOM’s six aims for improving health care quality]. Rockville (MD): Agency for Healthcare Research and Quality (US), 2008. Available: https://www.ncbi.nlm.nih.gov/books/NBK2677/table/ch4.t2/

¹⁷ Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (Redcap)–a Metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42: 377–81. doi:10.1016/j.jbi.2008.08.010

¹⁸ Harris PA, Taylor R, Minor BL, et al. The Redcap consortium: building an international community of software partners. J Biomed Inform 2019; 95: 103208. doi:10.1016/j.jbi.2019.103208

¹⁹ Dropbox [computer program]. San Francisco (CA) Dropbox;

²⁰ Power BI [computer program]. Seattle (WA) Microsoft;

²¹ Dedoose version 8.3.35, web application for managing, analyzing, and presenting qualitative and mixed method research data. [ program ]; 2020.