The WHO Global Benchmarking Tool: a game changer for strengthening national regulatory capacity

Abstract

Many low-income and middle-income countries lack the capacity to effectively and efficiently regulate medical products in their countries. To support countries in strengthening their capacity, WHO has developed the Global Benchmarking Tool (GBT) as the global standard for objectively assessing regulatory capacity for medicines and vaccines. The GBT is a game changer because it is the first globally accepted tool for assessing and strengthening national regulatory authorities. The inclusion of an institutional development plan in the GBT methodology provides context-specific actionable steps countries can take to advance their system’s functionality and maturity. The GBT facilitates coordination and improves the effectiveness of regulatory strengthening efforts. The tool also facilitates regulatory reliance and harmonisation, which helps to improve timely access to quality-assured medicines, and creates incentives for trade, particularly in countries and regions with a strong pharmaceutical manufacturing base. The GBT is a necessary tool for creating strong and effective regulatory systems, which are critical for ensuring the efficacy, safety and quality assurance of medicines and populations’ timely access to these medicines. In outlining the benefits of the GBT, this paper also offers some specific ideas for strengthening the GBT framework and process.

Correspondence to Dr Javier Guzman; [email protected]

Summary box

• Effective regulation of medical products is critical for ensuring access to safe, effective and quality-assured medical products in a well-functioning health system.

• WHO’s Global Benchmarking Tool (GBT) Revision VI is the first globally accepted tool for objectively assessing and strengthening regulatory capacity.

• The GBT provides countries with a systematic approach for strengthening their regulatory systems.

• The GBT fosters regulatory reliance and harmonisation, which increases timely access to quality-assured medical products and boosts pharmaceutical trade.

Introduction

National regulatory authorities (NRAs) play the critical role of ensuring the efficacy, safety and quality of medicines, both before and after products enter the market. However, WHO estimates that only 30% of NRAs among its member states have the capacity to effectively and efficiently regulate medical products in their countries.¹ Legal and regulatory frameworks are lacking or fragmented in some countries, which means the NRA may not have the mandate and authority to perform all regulatory functions.² Chronic underfunding often exacerbates challenges with human resource shortages and inadequate facilities. Without an effective and fully operational regulatory system, it is impossible to ensure that pharmaceutical products circulating in a country are safe, effective and of assured quality. WHO estimates that one-third of the world’s population lacks timely access to quality-assured medicines.³ Poor-quality medicines, often substandard or falsified, put patients at risk, fail to treat or prevent the disease for which they are intended, reduce the cost effectiveness of medicines and waste health systems resources.⁴ Approximately 3.75% of all under 5 deaths in 2013 were associated with the consumption of poor-quality antimalarials based on a sample of 39 sub-Saharan African countries.⁵

In response to the World Health Assembly’s Resolution 67.20 on regulatory system strengthening for medical products, WHO began developing the Global Benchmarking Tool (GBT) in 2014. The Resolution called for supporting member states in regulatory systems strengthening by using WHO tools to evaluate national regulatory systems, analyse evidence on regulatory system performance, and facilitate the formulation and implementation of institutional development plans (IDPs). The development of the GBT was an iterative process based on the mapping and evaluation of existing assessment tools and extensive stakeholder consultations with member states and other global stakeholders.⁶ The GBT Revision VI was finalised and released in 2018 and serves as the global standard for objectively assessing regulatory capacity for medicines and vaccines. It is anticipated that future iterations of the tool will yield GBT Plus and include blood, blood products and medical devices.

The GBT Revision VI is a game changer because it is the first globally accepted tool for objectively assessing and strengthening NRAs, but its benefits have not been widely communicated. This paper provides an overview of the tool and seeks to analyse some of its key benefits for countries. It also proposes some ideas for improving the GBT process.

Assessing and strengthening NRAS with a globally accepted tool

The benchmarking process includes a preassessment, self-assessment and formal benchmarking by WHO. The formal benchmarking process can take 2–5 years and consists of independent experts using the GBT factsheets produced by WHO and a computerised version of the tool as a rubric to assess a country’s NRA maturity level. The GBT uses 268 (sub)indicators disaggregated into nine indicator categories to measure capacity across an overarching framework (national regulatory system) and eight regulatory functions: registration and marketing authorisation, pharmacovigilance, market surveillance and control, licensing of establishments, regulatory inspections, laboratory testing, clinical trials oversight and lot release of vaccines (table 1). The benchmarking identifies strengths and weaknesses in the various regulatory functions and scores the system in terms of maturity level, ranging from 1 to 4. The levels correlate to no formal approach (level 1); reactive approach (level 2); stable, well-functioning system (level 3) and continual improvement emphasized (level 4). To date, at least 75 countries have completed self-assessments or formal benchmarks.

Table 1

Indicator categories by regulatory function

Indicator categories	Regulatory systems	Marketing authorisation	Vigilance	Market surveillance and control	Licensing	Regulatory inspections	Laboratory testing	Clinical trials	Lot release	Total
Leadership and crisis management	5									5
Legal provisions, regulations, and guidelines	9	13	7	7	5	5	2	11	2	61
Monitoring progress and assessing impact	2	2	2	3	2	5	4	4	4	28
Organisation and governance	4	2	2	2	2	2	2	2	2	20
Policy and strategic planning	5						7			12
Quality and risk management system	14									14
Regulatory process		10	8	8	4	6	6	7	3	52
Resources	12	4	4	4	4	4	6	4	4	46
Transparency, accountability and communication	9	4	3	3	2	4	1	2	2	30
Total	60	35	26	27	19	26	28	30	17	268

Key to the GBT process is the development of an IDP based on the identified strengths and weaknesses in capacity. Countries lead the formulation of their own IDPs, their implementation and monitoring of progress with support from development partners such as US Agency for International Development (USAID), the Bill & Melinda Gates Gates Foundation and the Global Fund. The IDP sets realistic standards and defines the interventions needed to progress on system maturity. Scores are not typically the same across all regulatory functions, and all functions do not need to be strengthened at the same level. Countries can, therefore, prioritise IDP interventions based on their national context, including epidemiological profile, local market characteristics and the extent to which they rely on other countries’ NRAs. Interventions are incremental, and their cumulative effect leads to progress on the maturity level. This means that the GBT process, including the IDP, provides each country with a tailored approach for strengthening its regulatory system.

Benefits of a common language and the power of measurement

The GBT creates a new opportunity for and a renewed interest in medicines regulation. It strengthens regulatory systems by promoting good regulatory practices and facilitating coordination of regulatory systems strengthening efforts. The tool also facilitates regulatory reliance and harmonisation, which provide both public health and economic benefits.

Strengthening regulatory systems

Using a system of maturity levels, the GBT provides a systematic approach for defining and strengthening regulatory capacity. Rather than relying on disease focused or vertical assessments and interventions, the GBT measures the various system functions that must work together to make regulatory activities effective and efficient. The tool clearly defines and promotes good regulatory practices. The assessment process helps authorities to improve their processes, particularly with respect to transparency. For example, the process requires NRAs to have open consultations and disseminate information (eg, publishing policies) to stakeholders. Countries are in the driver’s seat, so the process enables regulators and policy-makers to become more transparent, confident and self-reliant. Importantly, the formulation and implementation of the IDP provides actionable steps countries should take to strengthen their system’s functionality and maturity. The IDP helps to ensure consistency and protects the process from changes in senior management or government and from political interference. The maturity-level concept can be easily communicated to policy-makers, including the ministries of health and finance and the head of government, which can be critical for generating interest in regulatory systems strengthening.

The GBT’s common language also facilitates communication and coordination between countries and donors supporting the strengthening of NRAs. The coalition of interested parties (CIP), convened and coordinated by WHO, brings together the national government, donors and implementing partners to coordinate their priorities and resources for implementing a country’s IDP. The GBT enables this entire process as the various stakeholders share a common understanding of the NRA’s capacity, the standards needed to strengthen it, and how to measure that strengthened capacity. Donors can feel confident that their resources are not being wasted because the GBT provides a systematic approach for measuring and strengthening regulatory system capacity to a defined maturity level. The IDP provides the basis in the CIP for each partner to clearly map out and optimise its contribution to the process, thereby helping countries find the support needed to address the weaknesses in their regulatory systems. This eliminates duplication in technical assistance and makes the regulatory system strengthening process more effective and efficient, with donors having confidence in the appropriate use of their resources.

Regulatory reliance and harmonisation

The common language provided by the GBT facilitates regulatory reliance and harmonisation. Regulatory reliance refers to an NRA using work (eg, scientific assessments, regulatory decisions) shared by a reference authority, as defined by the GBT, to inform and perform its own regulatory functions.⁷ Reliance allows NRAs to work together to identify and use shared public goods (eg, dossier reviews, inspection outputs) and make the best use of their limited resources to focus on country-specific activities that must be done in-country, such as postmarketing surveillance. Regulatory harmonisation refers to the process by which regulatory requirements are unified and similar across participating NRAs in terms of legal instruments, technical standards, and guidelines. The GBT provides the standards and creates the trust needed for harmonisation initiatives to succeed. Several regional harmonisation initiatives exist, including those of the Regional Economic Communities in Africa—East African Community, Intergovernmental Authority on Development, Economic Community of West African States—and outside Africa including the Association of Southeast Asian Nations, the South-East Asia Regulatory Network and the Caribbean Community (CARICOM). Through collaboration and cooperation, weaker member states have learnt from more advanced NRAs within the region and received assistance with rapid benchmarking, thus improving their regulatory system.

Timely access to quality-assured medicines

Regulatory harmonisation and reliance allow countries to reap the public health benefits of having quality-assured essential medicines available in their markets and in a more timely manner. For example, reliance on reference authorities helps to overcome weak regulatory capacity for medicines registration. Medicines registration ensures the efficacy, safety, and quality assurance of medicines being introduced to a market. This process can be slow, burdensome and ineffective in poorly functioning NRAs and often becomes a superficial ‘rubber stamp’ or a barrier to timely access to quality-assured medicines. CARICOM, through its Caribbean Regulatory System (CRS), has relied on Pan American Health Organization (PAHO)-designated NRAs of Regional Reference (ie, Argentina, Brazil, Canada, Chile, Colombia, Cuba, Mexico and the USA), European Medicines Agency and the WHO prequalification programme to prioritise the registration of generic essential medicines. This reliance reduces the resources needed for medicines registration and shortens the process from years to months in the CARICOM region.⁸ It also means that already limited resources can be reallocated to other key regulatory functions such as pharmacovigilance to help ensure patient safety.

The ZaZiBoNa Initiative, a harmonisation effort among countries in the Southern Africa Development Community, has also facilitated sharing of resources and expertise to help streamline the medicines registration process and inspection across participating NRAs and improve access to medicines.⁹

Boost to pharmaceutical trade

In addition to their public health benefits, reliance and harmonisation mechanisms create multiple incentives for trade, particularly in countries and regions with a strong pharmaceutical manufacturing base. As low-income countries are graduated from global procurement mechanisms and middle-income countries increase their procurements in pursuit of universal health coverage (UHC), regional and national procurement is expected to increasingly require compliance with the standards of NRAs scored at maturity level 3. This has the potential to exclude many manufacturers from local and international bids. Countries with a strong manufacturing base, therefore, have an economic incentive to benchmark their NRAs. For example, Mexico undertook reforms in 2011 to optimise its regulatory body, Comisión Federal para la Protección contra Riesgos Sanitarios (COFEPRIS), which led to its designation as a PAHO reference authority in 2012 and recognition by WHO in 2014 as a level-3 authority that can sponsor authorised Mexican vaccines for WHO prequalification.¹⁰ It is estimated that strengthening COFEPRIS has contributed to a 13.2% growth in the local pharmaceutical market between 2011 and 2014.¹⁰

Additionally, manufacturers from countries with reference authorities often get preferential treatment in regional markets. In small states such as those in the CARICOM region, CRS’ streamlined process, facilitated by its reliance on PAHO-designated NRAs of Regional Reference, provides a single entry point and an abbreviated dossier review for products approved by the reference authorities. CRS has committed to issuing decisions on registration and marketing authorisation within 60 calendar days. Such accelerated and streamlined processes in regional economies can create incentives for manufacturers to enter the market and for their countries’ NRAs to maintain their reference status for the economic benefits. As more countries benchmark their NRAs, neighbouring countries will be encouraged to invest in regulatory systems strengthening. More broadly, through reliance and harmonisation, the GBT helps to build trust and promote confidence in medical products and benchmarked NRAs, which will boost the quality of local pharmaceutical manufacturing and regional exports.

Key priorities for moving forward

As more NRAs use the GBT, there are a few issues that WHO and its partners may need to address. GBT results are not publicly available, and publication of the results is left to the discretion of individual NRAs. Countries should be urged to be more transparent with their benchmarking results. The GBT process does not include a costing methodology for the IDP. Currently, costs assigned to the IDP are arbitrary, and countries would benefit from using a rigorous methodology for estimating the cost of implementing their IDPs. Given the costs and length of the GBT process, the incentive for countries to benchmark their NRAs may not always be apparent to policy makers and political stakeholders. Explicitly communicating the benefits related to public health and trade might help convince decision makers to prioritise regulatory systems strengthening efforts.

The maturity level of a national medicines regulatory authority might not provide adequate information on performance, such as quality of regulatory outcomes and manufacturers’ level of compliance. In 2018, WHO initiated the development of a framework for WHO Listed Authority (WLA).¹¹ The WLA framework will eventually replace the stringent regulatory authority concept, which is currently used in WHO’s prequalification programme and Global Fund procurements. This change in methodology aims to improve trust, reliance and transparency in regulatory systems, which in turn allows for regulatory resources to go further.¹¹ However, the proposed WLA framework is separate from the GBT and will require an assessment of NRA performance. Development of the WLA framework is ongoing and it remains uncertain the extent to which WHO will require maturity level 3 or 4 as an eligibility criterion.¹² The proposed WLA framework will likely expand the pool of regulatory authorities the WHO prequalification programme can engage through abridged procedures for prequalification listing, resulting in an automatic scale up of the programme and will more strongly facilitate global and regional reliance and harmonisation efforts.¹²

Conclusion

Strong and effective regulatory systems are essential for public health as they ensure access to safe, effective, quality-assured medical products and promote appropriate use. Regulatory systems, therefore, provide the enabling foundation for achieving UHC, a target of Sustainable Development Goal 3. The GBT offers a unique opportunity to the global health community because it answers the fundamental question of how to measure and strengthen regulatory capacity. Importantly, the tool shifts the focus solely from scoring the system to actionable steps and strategic investments for strengthening regulatory systems. Given the increasing complexity of pharmaceutical manufacturing and trade, NRAs will face increasing challenges in effectively and efficiently regulating medical products. The GBT promotes good regulatory practices and facilitates reliance, collaboration and harmonisation, all of which builds trust in NRAs and medical products and boosts pharmaceutical trade. By outlining these benefits, this paper hopes to prompt interest and investigation of the GBT and its role in regulatory systems strengthening. The paper also offers some specific ideas to further strengthen the GBT framework and process.

Footnote

Handling editor Seye Abimbola

Contributors All authors contributed equally to the conceptualisation and drafting of this manuscript.

Funding US Agency for International Development (7200AA18C00074).

Competing interests None declared.

Patient consent for publication Not required.

Provenance and peer review Not commissioned; externally peer reviewed.

Data availability statement All relevant data are included in the article and are publicly available.

References

¹ WHO. WHO essential medicines & health products annual report 2017: towards access 2030 [Internet]. Geneva, 2018. https://apps.who.int/iris/bitstream/handle/10665/272972/WHO-EMP-2018.01-eng.pdf?sequence=1&isAllowed=y

² Ndomondo-Sigonda M, Miot J, Naidoo S, et al. Medicines regulation in Africa: current state and opportunities. Pharmaceut Med 2017; 31: 383–97. doi:10.1007/s40290-017-0210-x http://www.ncbi.nlm.nih.gov/pubmed/29200865

³ Roth L, Bempong D, Babigumira JB, et al. Expanding global access to essential medicines: investment priorities for sustainably strengthening medical product regulatory systems. Global Health 2018; 14: 102. doi:10.1186/s12992-018-0421-2 http://www.ncbi.nlm.nih.gov/pubmed/30382856

⁴ WHO. A Study on the Public Health and Socioeconomic Impact of Substandard and Falsified Medical Products [Internet]. Geneva, 2017. https://www.who.int/medicines/regulation/ssffc/publications/SE-Study_EN_web.pdf?ua=1

⁵ Renschler JP, Walters KM, Newton PN, et al. Estimated under-five deaths associated with poor-quality antimalarials in sub-Saharan Africa. Am J Trop Med Hyg 2015; 92: 119–26. doi:10.4269/ajtmh.14-0725 http://www.ncbi.nlm.nih.gov/pubmed/25897068

⁶ WHO. WHO Medicines, Vaccines and Pharmaceuticals (MVP) 2018 annual report: promoting access to safe, effective, quality and affordable essential medical products for all [Internet]. Geneva, 2019. https://apps.who.int/iris/bitstream/handle/10665/324765/WHO-MVP-EMP-2019.03-eng.pdf?ua=1. (cited 2020 Jun 1).

⁷ Luigetti R, Bachmann P, Cooke E, et al. Collaboration, not competition: developing new reliance models. WHO Drug Inf 2016; 30: 558–66.

⁸ Preston C, Freitas Dias M, Peña J, et al. Addressing the challenges of regulatory systems strengthening in small states. BMJ Glob Health 2020; 5: e001912. doi:10.1136/bmjgh-2019-001912 http://www.ncbi.nlm.nih.gov/pubmed/32180997

⁹ Ndomondo-Sigonda M, Miot J, Naidoo S, et al. The African medicines regulatory harmonization initiative: progress to date. Med Res Arch 2018; 6. doi:10.18103/mra.v6i2.1668

¹⁰ Arriola Peñalosa MA, Cavazos Cepeda R, Alanis Garza M, et al. Optimized medical product regulation in Mexico: a win-win for public and economic health. Ther Innov Regul Sci 2017; 51: 744–50. doi:10.1177/2168479017701503 http://www.ncbi.nlm.nih.gov/pubmed/30227102

¹¹ WHO. Concept Note: A Framework for Evaluating and Publicly Designating Regulatory Authorities as WHO-Listed Authorities [Internet]. Geneva, 2019. https://www.who.int/medicines/areas/quality_safety/quality_assurance/qas19_808_WHO_listed_authorities.pdf?ua=1. (cited 2020 Jun 1).

¹² WHO. Policy. Evaluating and publicly designating regulatory authorities as WHO listed authorities. Draft for comments [Internet], 2019. Available: http://www.who.int/medicines/areas/quality_safety/quality_assurance/guidelines/en/