Benchmarking Drug Regulatory Systems for Capacity Building: An Integrative Review of Tools, Practice, and Recommendations

Background

Benchmarking, a common systematic practice that allows organizations to measure and compare key practice metrics to understand what, how and where changes are needed to improve performance, has been increasing employed on drug regulatory systems to achieve pharmaceutical system strengthening and universal health coverage.¹ Drug regulatory systems operated by national regulatory authorities (NRAs) play an integral role in the pharmaceutical system destined to ensure equitable access to essential medical products, vaccines and technologies of assured quality, safety, efficacy and cost-effectiveness, and their scientifically sound and cost-effective use.² The quest to excel the regulatory practice is further heightened amid the challenges brought about by, among other forces, the innovation and technology advancement, and major public health incidents.

By comparing their performance and capacities against a reference point, NRAs can determine how they perform, their weaknesses and strengths, and how to prioritise actions to continuously improve quality use of pharmaceutical products with respect to the local context.³ It is envisioned that benchmarking NRAs would also benefit the development of strategies for promoting regulatory practice standardization for transnational harmonization, reliance and recognition, as well as system resilience at country level in response to globalization of pharmaceutical products.^4,5

Drawing on the integrated approach of benchmarking and the experiences of public sector benchmarking,^6-9 the practices of benchmarking drug regulatory capacities are complicated. First, to achieve a meaningful outcome of benchmarking, it is imperative to have a realistic decision-making about the development endpoint of the NRA (whether it be a “gatekeeper” mitigating drug-related risks to the public health and/or as an “enabler” supporting research and innovation that weights on the different functionalities of the system).¹⁰ The complexity in operating benchmarking NRAs is further compounded by the different choices of reference points and tools for measuring functionalities, the means to collect data according to the predefined parameters, and the process of analysis in order to draw reliable comparison against the reference point. More importantly, it is the efforts put into learning and changes implementation based on evaluation results that matter in the quest for better capacities through benchmarking.

Given the growing interests in benchmarking drug regulatory systems, it becomes highly relevant to see what research and country experiences has had to say about this phenomenon. Previous literature mainly focused on a number of aspects of regulatory benchmarking including: the introduction of the practice of benchmarking NRA¹¹; the application of benchmarking NRAs for different purposes such as public health emergencies^12,13 and pharmacovigilance^14,15; various benchmarking tools or programmes such as the Global Benchmarking Tool (GBT) developed by the World Health Organization (WHO),¹⁶ and Optimizing Efficiencies in Regulatory Agencies (OpERA)¹⁷; and the country experiences of assessing and comparing regulatory capacities using benchmarking.^15,18-20 However, such literature is yet to be systematically analyzed and reviewed to depict an overall research landscape about how regulatory capacities can be approved through benchmarking NRAs. This is especially concerning when considering that, according to the WHO, 70% of its member states are not able to effectively and efficiently regulate medical products in their nations, especially in many low- and middle-income countries.^21,22

As such, this study aimed to answer the following questions: What are the scopes of regulatory capacities covered by benchmarking NRAs? What tools are available for benchmarking regulatory capacities? What benefits have the NRAs seen from benchmarking regulatory capacities? And what is the most recent development in the benchmarking practices? For the purpose of this study, in consultation with the resolution WHA 67.20 by the WHO, benchmarking of regulatory systems implies “a structured and documented process by which national drug regulatory authorities can identify and address gaps with the goal of reaching a level of regulatory oversight commensurate with a stable, well-functioning and integrated regulatory system.”²³

Methods and Materials

This study employed an integrative and critical review of the research on benchmarking drug regulatory capacities in compliance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines.²⁴ An integrative literature review was considered appropriate for the purpose of the study because it focused on combining, critiquing, and studying literature on the topic of benchmarking regulatory capacities in an integrated way in order to generate new frameworks and perspectives on this topic.²⁵

Search Strategy

Identification of the journal articles eligible for this review study consisted of the following steps. The primary concepts in this review included “benchmarking” and “regulatory capacity.” Based on preliminary research, the potential synonyms of “benchmarking” may include “benchmark,” “ranking,” “index,” “performance,” “indicator,” “evaluation,” and “assessment.” A pilot search for each term was conducted in PubMed and Web of Science to determine the frequency and relevance of each term. The four most frequent and relevant terms (“benchmark,” “ranking,” “index,” and “indicator”) were chosen to be included in the search strategy. Similarly, the terms “drug regulation,” “medicine regulation,” “regulatory authority,” “regulatory agency,” and “regulatory capacity” were used in the search strategy to reflect the concept of “regulatory capacity.” The search terms used in Chinese databases were: ((监管能力 OR 监管体系 OR 监管措施) AND (框架 OR 指标 OR 工具 OR 模型) AND (药品)).

Six databases (PubMed, Scopus, Medline, Web of Science, Science Direct, and China National Knowledge Infrastructure) were searched for eligible literature since the database inception till 30 November 2022. To ensure an effective search, Medical Subject Headings terms, synonyms and keywords related to the two concepts were used to develop a comprehensive search strategy. Terms of each concept were combined using OR, then the two concepts were combined using AND. In addition, reference lists and citations of included literature were screened to identify possibly eligible studies for inclusion.

Furthermore, international organizations and government agencies which had previously issued formal regulatory benchmarking tools or programmes were identified from the eligible studies and searched for eligible documents to be included in this study. The search covered the websites of such entities including the WHO (https://www.who.int/), the Organization for Economic Co-operation and Development (OECD) (https://www.oecd.org/), the Heads of Medicine Agencies (HMA) (https://www.hma.eu/), the Centre for Innovation in Regulatory Science (CIRS) (https://www.cirsci.org/), the US Government Accountability Office (GAO) (https://www.gao.gov/).

Inclusion and Exclusion Criteria

Literature, published in either English or Chinese, was included if it directly discussed about benchmarking drug regulatory capacities, or employed benchmarking approach to compare different regulatory systems for the purpose of identifying gaps and making improvement. Literature which investigated, evaluated or compared drug regulatory systems without employing benchmarking approach or posing any direct or indirect implications for benchmarking practice were excluded.

Screening Process

Literature was screened in compliance with the PRISMA statement. After removing the duplication, two authors (JS and XC) independently screened the titles and abstracts to identify literature that met the inclusion criteria. Full texts of potentially relevant articles were retrieved for detailed assessment. Discrepancies were discussed and resolved by agreement in consultation with 2 others authors (HH and COLU).

Data Extraction and Analysis

The following data was extracted from the included articles into an Excel table: title, authors, year of publication, paper type/study design, purpose, underlying evaluation tools or frameworks, key indicators, current problems, major findings, and major implications/comments. For eligible studies, 2 authors (JS and XC) independently extracted data and any disagreements were resolved by seeking confirmation from another author (COLU). Different tools or programmes collected were verified and presented into seven criteria categories of the NRAs for capacity building, including: the name of tool/programme, issuing organization and time, scope of application, purpose, focus areas, compositions and quality assessment methods (if applicable). Such information was then used for in-depth and comparative analyses.

Results

Literature Selection

Nine hundred and fifty-six records were identified from different databases, including 838 English initial records and 118 Chinese initial records. Upon removal of 468 duplicate articles, 488 records were proceeded to further screening by title and abstract, and, as a result, another 386 records were excluded. After full-text screening of the remaining 102 records, 59 records that did not related to benchmarking drug regulatory capacities were excluded. Ultimately, 43 eligible studies were included in this review (Figure 1). Furthermore, 6 documents about regulatory benchmarking tools or programmes retrieved from the websites of international organizations and government agencies were also included in this review for further analysis.

Close

Figure 1. PRISMA Flowchart of Study Selection. Abbreviation: PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analysis.

Literature Characteristics

The 43 articles and 6 additional records included in this review were published between 2005 and 2022. The description of 43 articles is presented in Table 1.^{11-15,18-20,26-60} The research design included literature review (n = 3),^28,32,49 expert interview (n = 1),⁴⁸ empirical analysis (n = 2),^43,52 comparative analysis (n = 7),^{12,15,35,42,47,58,59} retrospective analysis (n = 2),^19,41 questionnaire (n = 10),^{14,20,26,27,37-40,55,56} description analysis (n = 6),^{11,13,33,36,45,60} and mixed methods (n = 12).^{18,29-31,34,44,46,50,51,53,54,57}

As reported in 22 of the 43 included studies, benchmarking had been employed to assess the drug regulatory system in a specific country including developed countries — including UK (n = 1), Canada (n = 1), and Finland (n = 1) — and developing countries — including China (n = 11), Eritrea (n = 1), Myanmar (n = 1), Rwanda (n = 1), Turkish (n = 1), Zimbabwe (n = 1), Malaysia (n = 1), Iran (n = 1), and Brazil (n = 1). Comparing drug regulatory systems using benchmarking at regional level had also been reported for the Caribbean Community (CARICOM) region (n = 1), East Africa (n = 1), South Africa (n = 3), and West Africa (n = 1). Furthermore, cross-country benchmarking based on international organizations, such as Asia-Pacific Economic Cooperation (APEC) (n = 2) and CIRS (n = 1), were also reported in 15 studies.

In terms of benchmarking tools, 11 of the 43 included studies assessed the drug regulatory systems based on the GBT, of which 1 study combined indicators from the GBT and the Benchmarking of European Medicines Agencies (BEMA) (n = 1), 2 studies combined GBT indicators and OpERA tool (n = 2) and 2 studies combined GBT indicators and other indicators tool (n = 2). Four studies employed the OpERA tool to evaluated the NRAs’ regulatory capabilities and 14 studies employed other organizations evaluation indicators or methods. The remaining 16 studies used self-developed indicators or methods when conducting benchmarking.

The Main Themes of the Included Literature

After review and analysis of the included literature, 3 main themes related to benchmarking regulatory capabilities were identified: introduction of the concepts or methods of benchmarking; the application of benchmarking for internal assessment; and the application of benchmarking for external evaluation.

Introduction of the Concepts or Methods of Benchmarking

Four studies introduced benchmarking and highlighted the advantages and benefits of applying benchmarking methods or tools in the assessment of regulatory capability. For instance, among these 4 studies, 2 of them focused on explaining the GBT,^15,27 1 study emphasised on the utilization of three-party evaluation systems,³³ and 1 study explored the relationship between drug regulatory capabilities and regulatory performance.⁵³

The application of Benchmarking for Internal Assessment

A total of 19 studies were conducted to internally assess the regulatory capabilities of NRAs. Of these, 6 studies were dedicated to observing the establishment of the national-level regulatory system.^{18,30,35,50,54,57} Additionally, 5 studies were designed to analyse the drug review process.^{19,31,34,40,47} Two studies focused on measuring the effectiveness of capability building in healthcare,^43,60 while 3 studies concentrated on post-marketing surveillance,^51,52,55 particularly regarding the circulation and safety of drugs. One study aimed to enhance the vaccine regulatory system in China,³² another aimed to evaluate the level of adherence to good review practices (GRevP),⁴⁶ and 1 study aimed to identify the allocation of resources for capability building.⁴⁹

The Application of Benchmarking for External Evaluation

Drug regulatory capability of various countries or regions were investigated in 20 studies. At the national level, 4 studies aimed to evaluation the implementation of regulatory standards, including the Caribbean Regulatory System,³⁹ GRevP,⁵⁶ Certificate of pharmaceutical product,⁴² Pharmacovigilance (PV) system master file.¹⁴ Three studies aimed to evaluate the current regulatory situation evaluation and analyse the main challenge or problems faced by NRAs.^30,41,45 Additionally, the capacity required for effective regulatory decision-making process was also a topic of interest in 2 studies.^26,48 At the operation level, 7 studies were designed to identify the problems of the drug review process in the target area,^{20,29,31,38,44,58,59} of which four studies focused on the new drug registration process.^25,43,57,58 The remaining 4 studies aimed to assess the regulatory functionalities related to pharmacovigilance,^28,37 vaccines¹¹ and clinical trials.¹²

Tools Used for Benchmarking

According to the 6 additional records retrieved from WHO (n = 1),¹⁶ OECD (n = 1),⁶¹ HMA (n = 1),⁶² CIRS (n = 1),¹⁷ GAO (n = 2),^63,64 a total of 5 benchmarking tools were identified (Table 2). These included the Global GBT Revision VI,¹⁶ the indicators of Regulatory Policy and Governance (iREG),⁶¹ the BEMA,⁶² OpERA programme,¹⁷ and the report evaluated the Workforce planning and Scientific-integrity-related procedures and training.^63,64 The 5 tools or programs were issued from 2013 to 2022. The number of indicators included in these tools or programs range from 3 to 288.

The benchmarking tools or programmes identified in this study were developed or recognised by a variety of international organizations, third-party professional organizations, and independent government departments, each with different purposes and focuses. Particularly, the GBT was developed to assess the national regulatory frameworks in terms of regulatory functions, while iREG was applicable to the investigation of the processes in relation to national regulatory policy. The tools issued by OpERA and GAO focused more specifically on the review processes and procedures. In addition, HMA was operated to liaise the regulatory frameworks of pharmaceutical product under the European Union (EU) and European Medicines Agency (EMA), while BEMA aimed to advance the standards of regulatory practices for individual member state bodies focusing on benchmarking management systems, drug authorization, pharmacovigilance, and inspection services. Further description of the assessment tools or programmes is presented in Table 2.

Functions, Indicators, and Sub-indicators Covered by the Benchmarking Tools

The GBT measured 9 functions across an overarching national regulatory system framework and regulatory functions (including national regulatory system; registration and marketing authorization; pharmacovigilance; market surveillance and control; licensing of establishments; regulatory inspections; laboratory testing; clinical trials oversight; and lot release of vaccines) by using 9 indicator categories (legal provisions, regulations and guidelines; organization and governance; policy and strategic planning; leadership and crisis management; quality and risk management system; resources; regulatory process; transparency, accountability and commination; and monitoring progress and assessing outcomes and impact) outlining 268 sub-indicators.

The iREG indicators measured three key principles (including stakeholder engagement, regulatory impact analysis and ex post evaluation) using a total of 61 sub indicators. With BEMA, the key performance indicators and specific performance indicators were not separately listed but embedded in 14 sub-indicators of GBT and iREG. In the OpERA programme, the five indicators for evaluating the regulatory process of drug review and approval aligned with the GBT M06 (Mechanism in place to monitor regulatory performance and output), while the indicators for the workforce planning and scientific-integrity-related procedures and training were derived from the GBT RS06 (Human resources to perform regulatory activities) and RS10 (Mechanism in place to monitor regulatory performance and output).

An integrative analysis of the above-mentioned 5 benchmarking tools and programmes provided an overall landscape of the benchmarking scope. As shown in Figure 2, an integrated benchmarking framework comprised of a total of 12 functions (including 4 at system level and 8 at operation level), 9 indicator categories and 382 sub-indicators.

Close

Figure 2. The Commonalities Across the Regulatory Benchmarking Tools and Programmes. The indicator categories are connected to the corresponding functions by lines of the same colors. Indicator categories with more connecting lines are those used more often to measure the regulatory functions. Abbreviations: GBT, Global Benchmarking Tool; iREG, Indicators of Regulatory Policy and Governance; BEMA, Benchmarking of European Medicines Agencies; OpERA, Optimising Efficiencies in Regulatory Agencies; GAO, Government Accountability Office.

The commonalities across the regulatory benchmarking tools and programmes in terms of the functions and indicators covered are depicted in Figure 2. As shown, benchmarking “1. Regulatory system” at the system level and “5. Marketing authorization” at the operation level were of common interests to at least 3 benchmarking tools or programmes. Other functions such as “2. Stakeholder engagement,” “3. Regulatory impact assessment,” and “4. Ex post evaluation” at the system level and “11. Clinical trials” at the operation level were also of common interests to at least 2 benchmarking tools or programmes. Comparatively, the GBT was the most comprehensive benchmarking tool covering 9 out of the 12 functions.

With regards to the indicators, it is also worth noting that all the indicators identified from the selected benchmarking tools or programmes corresponded to the 9 indicator categories in the GBT. This is demonstrated by the lines connecting the functions and the GBT indicator categories as shown in Figure 2. When considering the number of connecting lines of each of the GBT indicator category, it is also noted that the GBT indicator categories “1. Legal provisions, regulations and guidelines,” “2. Organization and governance,” “6. Resources,” “7. Regulatory process,” “8. Monitor process and access outcomes & impact,” and “9. Transparency, accountability and communication” were used most often to measure different functions of a regulatory system. More detailed information about the indicator categories used to measure the 12 functions of is provided in Table S1 of Supplementary file 1. Importantly, it can be seen that the assessment of any function in a drug regulatory system is a complex evaluation approach involving not one but multiple dimensions of indicators. More detailed description of the functions, indicators and sub-indicators is provided in Table S2.

Key Functions and Indicators Employed in the Benchmarking Studies

Among the 43 included studies, 15 studies covered multiple regulatory functions^{11,13,15,18,20,27,32-34,42,44,45,50,54,57} while the remaining 28 studies focused on only 1 function when evaluating the regulatory capacities of NRAs.^{12,14,19,26,28-31,35-41,43,46-49,51-53,55,56,58-60} Apart from 6 of the studies^{11,13,18,27,32,42} which fully adopted the 9 GBT indicator categories when evaluating the functions of interest, the remaining studies adopted only some of the GBT indicator categories and sub-indicators whenever deemed relevant by the researchers. In 5 studies,^{12,15,29,33,34} a combination of indicators from different benchmarking tools or programmes were used as measurements of the regulatory capacities. More detailed description about the key functions and indicators employed in the benchmarking studies is provided Table S3.

Most Common Problems and Recommended Actions Based on Benchmarking Results

Among the 43 studies, some common problems or challenges in drug regulation had been repeatedly reported and, in some occasions, recommended actions based on the benchmarking results had been proposed accordingly. As shown in Table 3, there were 6 key aspects identified as the major areas of concern which included: legal provision; regulatory process; resources; cooperation and communication; and stakeholder engagement.

With respect to legal prevision, 6 studies identified four main types of regulatory issues including the lack of flexible regulatory policies or guidelines (n = 2),^50,51 pharmacovigilance systems (n = 2),^28,36 the framework of emergence preparedness (n = 1),¹² and a lack of quality management systems (n = 1).¹⁸ Correspondingly, 1 study recommended advancing the establishment of policy and legal framework,⁴⁰ while 10 studies focused on the promotion of practical strategies and guideline.^{12,13,27,29,31,32,38,50,53,56}

Regarding the regulatory processes, 17 studies identified 10 common problems, with a significant emphasis on the prolonged product review time (n = 6),^{19,31,40,41,44,46} and 8 studies also pointed to setting milestone for review time.^{19,29,31,35,37,41,44,46} Twenty-two studies sought to optimise the regulatory processes and practice,^{5,18-20,29,31-33,35,37,39-41,44,46,47,51,52,56-59} but did not provide specific implementation details.

The shortage of human resources,^{18,27,39,40,42} training or education^{18,29,39,47,48,53,57} financial resources^42,49 and equipment or technique resources^18,29 were reported to exert a negative impact on regulation. In addition, the lack of transparency and communication,^14,29,31 as well as the lack of involvement of key stakeholders,³⁷ were identified as common problems upon benchmarking. Nineteen studies proposed enhancing communication and cooperation at all levels^{12,14,26,28,29,31,32,35,38,39,42-46,48,52,56,60} while 5 studies encouraged stakeholder participation to drive regulatory decision-making.^{14,15,28,38,48}

Moreover, 1 study mentioned the gaps between the academic outcomes of publications in peer reviewed journals or successful grant applications and the resolution of regulatory practices.⁶⁰ Nevertheless, it is worth noting that not every common problem was addressed with specific recommended action.

Anticipated Outcomes of Benchmarking Reported in the Included Studies

Eighteen studies assessed the outcomes when applying benchmarking for regulatory improvement, including the promotion of regulatory reliance and harmonization (n = 7),^{11,13,14,28,35,39,45} the enhancement of regulatory transparency (n = 4),^27,29,31,44 the reducing of timelines and improving patients’ access to new medicines (n = 3),^11,29,32 the optimization of publicly available information (n = 3),^27,44,52 and the improvement of pharmaceutical trade (n = 1).¹¹

Discussion

This literature review reaffirms that benchmarking has been employed by many NRAs as an important strategy in quality monitoring and management in pursuit of improvement in regulatory capacities. Further analysis of the included literature has depicted an overall research landscape on this phenomenon covering the main purposes of benchmarking, different benchmarking tools and comparison of the corresponding indicators, key indicators selected for benchmarking, major areas of improvement based on benchmarking results, most common recommended actions following up benchmarking practices, and anticipated of benchmarking outcomes. An integrative analysis of such findings gave rise to a framework for decision-makers in NRAs when deciding why and how benchmarking should be undertaken (Figure 3) which will be further discussed in the following. Nevertheless, the literature included in this review did not provide any empirical findings showing that NRAs had benefited from benchmarking regulatory capacities.

Close

Figure 3. Decision-Making Framework for Benchmarking Drug Regulatory Capacities. Abbreviations: WHO, World Health Organization; GBT, Global Benchmarking Tool; HMA, Heads of Medicine Agencies; BEMA, Benchmarking of European Medicines Agencies; OECD, Organization for Economic Co-operation and Development; iREG, Indicators of Regulatory Policy and Governance; CIRS, Centre for Innovation in Regulatory Science; OpERA, Optimizing Efficiencies in Regulatory Agencies; FDA, Food and Drug Administration; GAO, Government Accountability Office.

The Decision-Making Framework of Benchmarking

Using benchmarking to guide the advancement of pharmaceutical regulatory system echoed with the increasing emphasis of policy-makers on “evidence-informed health policy-making” (EIHP) to inform the decision-making in the contemporary healthcare.^65,66 The EIHP approach aims to fully inform the best available research evidence as an input to the healthcare policy-making process. As shown in the current review, more and more NRAs currently include the benchmarking results as one of evidence in their regulatory capacity monitoring and management.

As shown in Figure 3, the decision-making about benchmarking regulatory capacities involved multifaceted considerations of the benchmarking scope in terms of functionalities, the choice of benchmarking tools and reference points which would collectively determine the benchmarking outcome. At the functionality level, the benchmarking scope may encompass “strategy” (focus on the strategic goals and leadership of advancing NRAs such as the vaccine regulatory system³²), “operation” (focus on the key processes to eliminate the weaknesses of regulation such as the PV system,^14,28,36,38 “performance” (focus on the key performance indicators such as the delays in assessing applications owing to the staff manpower⁴⁰) and “training” (focus on priority areas of capacity building such as workforce planning and training^17,29). Another perspective when deciding on the scope might refer to pre-marketing and post-marketing functionalities of an NRA.

The next important consideration when conducting benchmarking in drug regulation practice is the choice of reference points. When benchmarking was conducted internally, previous benchmarking results could be used as a baseline for continuous monitoring of performance to identify any changes in regulatory practice over time. Cross-country benchmarking, on the other hand, could be used to inform actions for regulatory practice standardization that promotes collaboration and harmonization across a region or a consortium. Coordinated efforts across NRAs in improving regulatory practice in common areas have been recognised as an important measure to facilitate regulatory reliance and harmonization at regional level.^11,13,39,45

It was also found that benchmarking was conducted with different tools and indicators which were selected based on the functionalities of interests. A range of benchmarking tools have been made available and it remained at decision-maker’s discretion about the choice of tools and the combination of indicators from readily available benchmarking tools or self-developed initiative as they were deemed fit the purpose of benchmarking.

Comprehensive benchmarking against an international benchmarking framework would further benefit the credibility and international recognition of the NRAs. For instance, following Singapore which reached Maturity Level 4 in medicines in February 2022 which is the highest level achievable for regulatory system evaluation against the WHO’s GBT, the Ministry of Food and Drug Safety of the Republic of Korea also announced in November 2022 that it had reached Maturity Level 4, in both medicines and vaccines regulations.⁶⁷ Countries with Maturity Level 3 or Maturity Level 4 according to the GBT are eligible to become a WHO listed authority so that they may be considered as a reference point by other regulatory authorities for reaching own decisions in approving medical products.⁶⁸ For NRAs to be evaluated and recognised as operating at an advanced level of performance with continuous improvement is pertained to profound both practical and signifying implications at an international level.⁶⁷ Benchmarking against best practices would further help NRAs to achieve regulatory excellence.

Benchmarking as a Process From Bench-Learning to Bench-Action

It is worth noting that while benchmarking is important in identifying gaps and weakness, it is the “bench-learning” and “bench-action” that are key to making changes.^69,70 However, little has been reported about communicating the aftermaths of benchmarking exercises and how to divert the findings and knowledge between and among researchers and NRAs to advance performance and address the gaps. All the included studies rested on the monologues about the relevance of benchmarking.

Decisions about employing benchmarking requires systematic planning and multifaceted perspectives (including but not limited to the political environment, the latest advances in pharmaceutical research and development, the unmet needs of the patients, the availability of high-level engagement and resources, etc) to formulate effective implementation approaches. Addressing the gaps in regulatory performance is part of a highly complex undertaking involving not just NRAs but also other counterparts in the pharmaceutical system, as well as other counterparts in the larger environment of health system, both locally and beyond.⁷¹ However, it appeared in this review that benchmarking often took place in silos with no significant engagement of researchers or other government agencies with the NRAs on following up the findings and solving critical gaps.

Indeed, as revealed in this review, barriers to implementing changes might be multifaceted encompassing, just to name a few, the political environment, the deficit in the information systems, the scarcity of related research to form the scientific foundation, and the lack of continuous engagement of leadership. Decisions about interventions that bring changes to drug regulatory practices often warrants the guidance from high-level governance to identify effective approaches and thus systematic planning.

To move this facet of practice forward, a systems thinking approach guided by implementation science might offer a roadmap that help translate the benchmarking findings into formulation of actions for facilitating changes. Systems thinking is an approach that advocates for the involvement of key stakeholders to map the drug regulatory system, identify where the key impediments lie, and design synergistic and system-ready pathway towards benchmarking practice.⁷² This approach calls for transdisciplinary and translational approaches and encourages relationship-building across various functionalities of the NRAs so as to achieve a common set of relevant goals and objectives on drug regulation.

Identifying the strengths and weaknesses in drug regulatory capacities through benchmarking using “systems thinking” approach then leads to the need for a coordinated and collaborative effort to implement and sustain changes in regulatory measure. Nevertheless, major challenges for bench-actions to be translated into sustained routine practice are foreseeable. There needs to be a scientific approach to identify the range of factors that are likely to facilitate the uptake of recommended actions and changes in regulatory practice, and to plan and act accordingly. More importantly, regulatory management systems to measure changes and demonstrate any outcomes associated with changes in regulatory capacities related to benchmarking is essential to support the sustainable development of the intervention or service. For this, implementation science knowledge and strategies must be employed and incorporate into the regulatory management systems to promote intervention validity, while collecting the data necessary for establishing evidence-based improvement to bargain for continuous resources input for benchmarking exercises.⁷³

Limitations of the Study Findings

This study provides a comprehensive view of benchmarking the NRAs for capacity building in term of the existing tools, practices, and recommendations based on literature retrieval, analysis and data synthesis. Nevertheless, our review has some limitations. The first limitation is that we were not able to define a uniform system to determine the “maturity level” for all indicators due to the heterogenicity of the assessment methods employed in each benchmarking tool or programme included in this study. For each indicator of regulatory capacity, the maturity level is important not only for reflecting the status but also for measuring progress. Drawing on the successful experiences of the work on the GBT by the WHO, future research is warranted to yield specific criteria of quality assessment methods for each indicator. Another limitation of this review may be contributed by publication bias. The risks of negative outcomes about benchmarking regulatory capacity being rarely or unlikely to be fully reported in the literature cannot be ruled out possibly compromising the comprehensiveness of the overall research landscape about regulatory benchmarking presented in this review. Furthermore, considering that not all benchmarking tools or programs, and the related regulatory performance data are publicly available, the limitation in full access to all regulatory benchmarking information may inevitably affect the completeness of the findings reported in this review.

Conclusion

Benchmarking drug regulatory capacities is a complex process that has been increasingly adopted by NRAs for measuring the regulatory performance and monitoring the progress. This review has analysed in detail the “why” and “how” to employ benchmarking to improve regulatory practice. For effective benchmarking that leads to bench-learning and bench-action, well-informed decisions about the goals, the scope, the choice of reference points and benchmarking tools are essential to guide the implementation strategies, coordination of resources, and stakeholders’ participation and cooperation. Nevertheless, the evidence for the possible benefits of benchmarking remains scarce. There is a need for more empirical studies to develop evidence about how benchmarking can improve drug regulatory capacities.

Ethical issues

Not applicable.

Competing interests

Authors declare that they have no competing interests.

Funding

This work was supported by a grant from University of Macau (reference numbers: SRG2021-00007- ICMS and MYRG2022-00229-ICMS).

Supplementary files

References

1. Khadem Broojerdi A, Baran Sillo H, Ostad Ali Dehaghi R, Ward M, Refaat M, Parry J. The World Health Organization global benchmarking tool an instrument to strengthen medical products regulation and promote universal health coverage. Front Med (Lausanne) 2020; 7:457. doi: 10.3389/fmed.2020.00457 [Crossref] [ Google Scholar]
2. World Health Organization (WHO). Everybody’s Business--Strengthening Health Systems to Improve Health Outcomes: WHO’s Framework for Action. WHO; 2007.
3. Lodge M. Regulatory capacity. In: Lodge M, Wegrich K, eds. The Problem-Solving Capacity of the Modern State: Governance Challenges and Administrative Capacities. Oxford University Press; 2014;63-85.
4. Hafner T, Walkowiak H, Lee D, Aboagye-Nyame F. Defining pharmaceutical systems strengthening: concepts to enable measurement. Health Policy Plan 2017; 32(4):572-584. doi: 10.1093/heapol/czw153 [Crossref] [ Google Scholar]
5. Haldane V, De Foo C, Abdalla SM. Health systems resilience in managing the COVID-19 pandemic: lessons from 28 countries. Nat Med 2021; 27(6):964-980. doi: 10.1038/s41591-021-01381-y [Crossref] [ Google Scholar]
6. Zairi M. Benchmarking: the best tool for measuring competitiveness. Benchmarking for Quality Management & Technology 1994; 1(1):11-24. doi: 10.1108/14635779410056859 [Crossref] [ Google Scholar]
7. Ahmed PK, Rafiq M. Integrated benchmarking: a holistic examination of select techniques for benchmarking analysis. Benchmarking for Quality Management & Technology 1998; 5(3):225-242. doi: 10.1108/14635779810234802 [Crossref] [ Google Scholar]
8. Tillema S. Public sector benchmarking and performance improvement: what is the link and can it be improved?. Public Money Manag 2010; 30(1):69-75. doi: 10.1080/09540960903492414 [Crossref] [ Google Scholar]
9. Triantafillou P. Benchmarking in the public sector: a critical conceptual framework. Public Adm 2007; 85(3):829-846. doi: 10.1111/j.1467-9299.2007.00669.x [Crossref] [ Google Scholar]
10. Dorsch JJ, Yasin MM. A framework for benchmarking in the public sector. Int J Public Sect Manag 1998; 11(2/3):91-115. doi: 10.1108/09513559810216410 [Crossref] [ Google Scholar]
11. Guzman J, O’Connell E, Kikule K, Hafner T. The WHO Global Benchmarking Tool: a game changer for strengthening national regulatory capacity. BMJ Glob Health 2020; 5(8):e003181. doi: 10.1136/bmjgh-2020-003181 [Crossref] [ Google Scholar]
12. Owusu Sekyere S, Škrnjug-Yudov I, Ateba Ngoa U. Leveraging WHO’s Global Benchmarking Tool to strengthen capacity in clinical trials oversight for public health emergencies: the GHPP VaccTrain model. Global Health 2022; 18(1):63. doi: 10.1186/s12992-022-00854-0 [Crossref] [ Google Scholar]
13. Khadem Broojerdi A, Alfonso C, Ostad Ali Dehaghi R, Refaat M, Sillo HB. Worldwide assessment of low- and middle-income countries’ regulatory preparedness to approve medical products during public health emergencies. Front Med (Lausanne) 2021; 8:722872. doi: 10.3389/fmed.2021.722872 [Crossref] [ Google Scholar]
14. Lavery C, Emmott J, Jeck-Thole S. An industry survey on managing the pharmacovigilance system master file in a global environment: the need for a pragmatic approach. Pharmaceut Med 2022; 36(4):233-245. doi: 10.1007/s40290-022-00422-2 [Crossref] [ Google Scholar]
15. Barry A, Olsson S, Minzi O. Comparative assessment of the national pharmacovigilance systems in East Africa: Ethiopia, Kenya, Rwanda and Tanzania. Drug Saf 2020; 43(4):339-350. doi: 10.1007/s40264-019-00898-z [Crossref] [ Google Scholar]
16. World Health Organization (WHO). WHO Global Benchmarking Tool (GBT) for Evaluation of National Regulatory Systems of Medical Products: Revision VI. WHO; 2021.
17. Centre for Innovation in Regulatory Science (CIRS). Measuring Process and Performance in Regulatory Agencies: The OpERA Programme. CIRS; 2020. https://www.cirsci.org/wp-content/uploads/2020/02/CIRS-RD-Briefing-74-OpERA-programme.pdf. Accessed October 2, 2020.
18. Shabani JBB, Kayitare E, Nyirimigabo E. The capacity of young national medicine regulatory authorities to ensure the quality of medicines: case of Rwanda. J Pharm Policy Pract 2022; 15(1):90. doi: 10.1186/s40545-022-00492-2 [Crossref] [ Google Scholar]
19. Sani NM, McAuslane N, Kasbon SH, Ahmad R, Yusof FAM, Patel P. An evaluation of Malaysian regulatory process for new active substances approved in 2017 using the OpERA methodology. Ther Innov Regul Sci 2020; 54(5):1215-1224. doi: 10.1007/s43441-020-00140-4 [Crossref] [ Google Scholar]
20. Rahalkar H, Sheppard A, Salek S. Comparison of BRICS-TM countries’ biosimilar regulatory frameworks with Australia, Canada and Switzerland: benchmarking best practices. Front Pharmacol 2021; 12:711361. doi: 10.3389/fphar.2021.711361 [Crossref] [ Google Scholar]
21. World Health Organization (WHO). WHO Benchmarks for International Health Regulations (‎IHR)‎ Capacities. WHO; 2019. https://apps.who.int/iris/handle/10665/311158. Accessed December 1, 2022.
22. World Health Organization (WHO). WHO Essential Medicines and Health Products: Annual Report 2017: Towards Access 2030. WHO; 2018. https://apps.who.int/iris/handle/10665/272972. Accessed December 1, 2022.
23. World Health Organization (WHO). Regulatory System Strengthening for Medical Products. Proceedings of the 67th World Health Assembly.2014:28.
24. Subirana M, Solá I, Garcia JM, Gich I, Urrútia G. A nursing qualitative systematic review required MEDLINE and CINAHL for study identification. J Clin Epidemiol 2005; 58(1):20-25. doi: 10.1016/j.jclinepi.2004.06.001 [Crossref] [ Google Scholar]
25. Torraco RJ. Writing integrative literature reviews: guidelines and examples. Hum Resour Dev Rev 2005; 4:356-367. doi: 10.1177/1534484305278283 [Crossref] [ Google Scholar]
26. Bujar M, McAuslane N, Walker S, Salek S. A process for evaluating quality decision-making practices during the development, review and reimbursement of medicines. Int J Health Policy Manag 2022; 11(2):128-137. doi: 10.34172/ijhpm.2020.86 [Crossref] [ Google Scholar]
27. Chaw EE, Suntornsuk L. Regulatory landscape analysis of Myanmar Food and Drug Administration based on the World Health Organization Global Benchmarking Tool. Pharm Sci Asia 2022; 49(5):498-505. doi: 10.29090/psa.2022.05.22.174 [Crossref] [ Google Scholar]
28. Garashi HY, Steinke DT, Schafheutle EI. a systematic review of pharmacovigilance systems in developing countries using the WHO pharmacovigilance indicators. Ther Innov Regul Sci 2022; 56(5):717-743. doi: 10.1007/s43441-022-00415-y [Crossref] [ Google Scholar]
29. Keyter A, Salek S, Banoo S, Walker S. A proposed regulatory review model to support the South African Health Products Regulatory Authority to become a more efficient and effective agency. Int J Health Policy Manag 2022; 11(6):795-809. doi: 10.34172/ijhpm.2020.213 [Crossref] [ Google Scholar]
30. Mahdavi M, Sajjadi Khasraghi J, Sajadi HS. Developing framework and strategies for capacity building to apply evidence-informed health policy-making in Iran: mixed methods study of SAHSHA Project. Int J Health Policy Manag 2022; 11(10):2236-2247. doi: 10.34172/ijhpm.2021.142 [Crossref] [ Google Scholar]
31. Sithole T, Salek S, Mahlangu G, Walker S. Comparison of the registration process of the medicines control authority of Zimbabwe with Australia, Canada, Singapore, and Switzerland: benchmarking best practices. Expert Rev Clin Pharmacol 2022; 15(1):109-119. doi: 10.1080/17512433.2022.1987883 [Crossref] [ Google Scholar]
32. Xing LY, Lin XN, Zhou HX, Mo JL, Chen M. Research on the construction of vaccine supervision quality management of drug regulatory authorities in compliance with WHO Global Benchmarking Tool (GBT). Chin Pharm 2022;36(4):391-398. [Chinese].
33. Zhang Q, Xie Y, Sheng LL, et al. Third-party evaluation methods and systems for drug regulatory measures. Chin food & Drug Administration Magazine. 2022;4:20-29. [Chinese].
34. Li F. Study on ability evaluation model for drug review authority. Chin J New Drugs 2021;30(24):2242-2248. [Chinese].
35. Rodier C, Bujar M, McAuslane N, Patel P, Liberti L. Use of the Certificate for Pharmaceutical Products (CPP) in 18 maturing pharmaceutical markets: comparing agency guidelines with company practice. Ther Innov Regul Sci 2021; 55(1):118-128. doi: 10.1007/s43441-020-00196-2 [Crossref] [ Google Scholar]
36. Russom M, Bahta I, Debesai M. Eritrean pharmacovigilance system: key strategies, success stories, challenges and lessons learned. Drug Saf 2021; 44(10):1021-1032. doi: 10.1007/s40264-021-01102-x [Crossref] [ Google Scholar]
37. Sithole T, Mahlangu G, Salek S, Walker S. Evaluation of the regulatory review process in Zimbabwe: challenges and opportunities. Ther Innov Regul Sci 2021; 55(3):474-489. doi: 10.1007/s43441-020-00242-z [Crossref] [ Google Scholar]
38. Hartmann K, Pagliusi S, Precioso A. Landscape analysis of pharmacovigilance and related practices among 34 vaccine manufacturers’ from emerging countries. Vaccine 2020; 38(34):5490-5497. doi: 10.1016/j.vaccine.2020.06.016 [Crossref] [ Google Scholar]
39. Keyter A, Salek S, McAuslane N, Banoo S, Azatyan S, Walker S. Implementation of a framework for an abridged review using good reliance practices: optimising the medicine regulatory review process in South Africa. Ther Innov Regul Sci 2020; 54(5):1199-1207. doi: 10.1007/s43441-020-00144-0 [Crossref] [ Google Scholar]
40. Liberti L, Extavour RM, Patel P, McAuslane N. An evaluation of the Caribbean regulatory system centralized assessment process for medicines submitted 2017-2018 using the OpERA methodology. J Pharm Policy Pract 2020; 13:56. doi: 10.1186/s40545-020-00261-z [Crossref] [ Google Scholar]
41. Patel P, Cerqueira DM, Santos GML. A baseline analysis of regulatory review timelines for ANVISA: 2013-2016. Ther Innov Regul Sci 2020; 54(6):1428-1435. doi: 10.1007/s43441-020-00169-5 [Crossref] [ Google Scholar]
42. Preston C, Freitas Dias M, Peña J, Pombo ML, Porrás A. Addressing the challenges of regulatory systems strengthening in small states. BMJ Glob Health 2020; 5(2):e001912. doi: 10.1136/bmjgh-2019-001912 [Crossref] [ Google Scholar]
43. Saaristo V, Hakamäki P, Koskinen H, Wiss K, Ståhl T. The comparative and objective measurement of health promotion capacity-building: from conceptual framework to operationalization. Glob Health Promot 2020; 27(1):24-32. doi: 10.1177/1757975918769608 [Crossref] [ Google Scholar]
44. Keyter A, Salek S, Banoo S, Walker S. The South African medicines control council: comparison of its registration process with Australia, Canada, Singapore, and Switzerland. Front Pharmacol 2019; 10:228. doi: 10.3389/fphar.2019.00228 [Crossref] [ Google Scholar]
45. Chong SSF, Lim JCW, Tominaga T. Developing key performance indicators to measure the progress of regional regulatory convergence and cooperation in Asia-Pacific Economic Cooperation (APEC). AAPS Open 2018; 4(1):4. doi: 10.1186/s41120-018-0024-2 [Crossref] [ Google Scholar]
46. Mashaki Ceyhan E, Gürsöz H, Alkan A, Coşkun H, Koyuncu O, Walker S. The Turkish medicines and medical devices agency: comparison of its registration process with Australia, Canada, Saudi Arabia, and Singapore. Front Pharmacol 2018; 9:9. doi: 10.3389/fphar.2018.00009 [Crossref] [ Google Scholar]
47. Tang WY, Zhang H, Jin DZ. Study on construction of knowledge management system for drug evaluation and inspection. Shanghai Med Pharm J 2018;39(13):56-57. [Chinese].
48. Li R, Ruiz F, Culyer AJ, Chalkidou K, Hofman KJ. Evidence-informed capacity building for setting health priorities in low- and middle-income countries: a framework and recommendations for further research. F1000Res 2017; 6:231. doi: 10.12688/f1000research.10966.1 [Crossref] [ Google Scholar]
49. Mery G, Dobrow MJ, Baker GR, Im J, Brown A. Evaluating investment in quality improvement capacity building: a systematic review. BMJ Open 2017; 7(2):e012431. doi: 10.1136/bmjopen-2016-012431 [Crossref] [ Google Scholar]
50. Liu P, Liu ZP. Measurement and Assessment of Regulatory Capability of Social Regulatory Agencies -Based on a comparative analysis of three regulatory authorities. Wuhan Univer J 2016;69(4):32-40. [Chinese].
51. Zhang M, Guan X, Li J, Gao H, Liu D, Shi L. Establishment of drug safety performance evaluation indicator system for work quality evaluation of drug administration department. Chin Pharm 2016;27(1):1-4. [Chinese].
52. Chen YC, Wang SZ, Jiang JF. Research on evaluation system of regulatory standards of circulation safety about essential medicines in rural in Jiangxi province. Chin Health Service Management 2015;5:373-377. [Chinese].
53. Yang S, Yang DS, Zheng XY, Huang G, Cong LL, Jiang B. Introduction to drug regulatory capabilities: a theory and framework. J Chin Pharm Sci 2015;24(4):257-262. [Chinese].
54. Yao JC, Zeng LG, Li YB, Hu L, Du J, Rao J. Research on the key indicators of drug regulation. Chin Pharmaceut. 2014,23(12):3-5. [Chinese].
55. Zhang X, Lin ZJ, Dong L. Questionaire designing of evaluation on drug safety supervision capacities based upon a 360 degree performance model. Chin Pharmaceut 2014;28(7):700-705. [Chinese].
56. Liu LL, McAuslane N, Tzou MC. Characterizing good review practices: a survey report among agencies of APEC member economies. Ther Innov Regul Sci 2013; 47(6):678-683. doi: 10.1177/2168479013494394 [Crossref] [ Google Scholar]
57. Yang S, Wang GP, Huang G, Jiang B. Research on evaluation index system of China’s drug regulatory capability. Chin Pharmceut J 2012;47(14):1175-1178. [Chinese].
58. McAuslane N, Cone M, Collins J, Walker S. Emerging markets and emerging agencies: a comparative study of how key regulatory agencies in Asia, Latin America, the Middle East, and Africa are developing regulatory processes and review models for new medicinal products. Drug Inf J 2009; 43(3):349-359. doi: 10.1177/009286150904300314 [Crossref] [ Google Scholar]
59. Hirako M, McAuslane N, Salek S, Anderson C, Walker S. A comparison of the drug review process at five international regulatory agencies. Ther Innov Regul Sci 2007; 41(3):291-308. doi: 10.1177/009286150704100302 [Crossref] [ Google Scholar]
60. Cooke J. A framework to evaluate research capacity building in health care. BMC Fam Pract 2005; 6:44. doi: 10.1186/1471-2296-6-44 [Crossref] [ Google Scholar]
61. Arndt C, Baker AC, Querbach T, Schultz R. 2015 Indicators of Regulatory Policy and Governance: Design, Methodology and Key Results. In: OECD Regulatory Policy Working Papers. Paris: OECD Publishing; 2015. 10.1787/5jrnwqm3zp43-en.
62. Heads of Medicines Agencies (HMA). Benchmarking of European Medicines Agencies (BEMA). Final Report on the Fourth BEMA Cycle. HMA; 2019. https://www.hma.eu/fileadmin/dateien/Human_Medicines/01-About_HMA/Working_Groups/BEMA/2019_10_BEMA_IV_Final_Report.pdf. Accessed December 1, 2022.
63. Government Accountability Office (GAO). FDA Workforce: Agency-Wide Workforce Planning Needed to Ensure Medical Product Staff Meet Current and Future Needs. GAO; 2022. https://www.gao.gov/products/gao-22-104791. Accessed December 1, 2022.
64. Government Accountability Office (GAO). Scientific Integrity: HHS Agencies Need to Develop Procedures and Train Staff on Reporting and Addressing Political Interference. GAO; 2022. https://www.gao.gov/assets/gao-22-104613.pdf. Accessed December 1, 2022.
65. Bowen S, Zwi AB. Pathways to “evidence-informed” policy and practice: a framework for action. PLoS Med 2005; 2(7):e166. doi: 10.1371/journal.pmed.0020166 [Crossref] [ Google Scholar]
66. Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 1: what is evidence-informed policymaking?. Health Res Policy Syst 2009; 7(Suppl 1):S1. doi: 10.1186/1478-4505-7-s1-s1 [Crossref] [ Google Scholar]
67. World Health Organization (WHO). Republic of Korea Achieves the Highest Who Level for Regulation of Medicines and Vaccines. WHO; 2022. https://www.who.int/news/item/29-11-2022-republic-of-korea-achieves-the-highest-who-level--for-regulation-of-medicines-and-vaccines. Accessed December 1, 2022.
68. World Health Organization (WHO). Manual for Benchmarking of the National Regulatory System of Medical Products and Formulation of Institutional Development Plans. WHO; 2021.
69. Krutten J. Benchmarking in the pharmaceutical industry. Mark Health Serv 1999; 19(3):14-22. [ Google Scholar]
70. Zairi M, Ahmed PK. Benchmarking maturity as we approach the millennium?. Total Qual Manag 1999; 10(4-5):810-816. doi: 10.1080/0954412997848 [Crossref] [ Google Scholar]
71. Peters DH. The application of systems thinking in health: why use systems thinking?. Health Res Policy Syst 2014; 12:51. doi: 10.1186/1478-4505-12-51 [Crossref] [ Google Scholar]
72. Naaldenberg J, Vaandrager L, Koelen M, Wagemakers AM, Saan H, de Hoog K. Elaborating on systems thinking in health promotion practice. Glob Health Promot 2009; 16(1):39-47. doi: 10.1177/1757975908100749 [Crossref] [ Google Scholar]
73. Seaton TL. Dissemination and implementation sciences in pharmacy: a call to action for professional organizations. Res Social Adm Pharm 2017; 13(5):902-904. doi: 10.1016/j.sapharm.2017.05.021 [Crossref] [ Google Scholar]