Comparing 3 Approaches for Making Vaccine Adoption Decisions in Thailand

Background: The World Health Organization (WHO) has developed the Total System Effectiveness (TSE) framework to assist national policy-makers in prioritizing vaccines. The pilot was launched in Thailand to explore the potential use of TSE in a country with established governance structures and accountable decision-making processes for immunization policy. While the existing literature informs vaccine adoption decisions in GAVI-eligible countries, this study attempts to address a gap in the literature by examining the policy process of a non-GAVI eligible country. Methods: A rotavirus vaccine (RVV) test case was used to compare the decision criteria made by the existing processes (Expanded Program on Immunization [EPI], and National List of Essential Medicines [NLEM]) for vaccine prioritization and the TSE-pilot model, using Thailand specific data. Results: The existing decision-making processes in Thailand and TSE were found to offer similar recommendations on the selection of a RVV product. Conclusion: The authors believe that TSE can provide a well-reasoned and step by step approach for countries, especially low- and middle-income countries (LMICs), to develop a systematic and transparent decision-making process for immunization policy.


Background
Traditionally, ministries or departments of health had sole responsibility for health service policy at the national level. These entities oversaw agenda setting, policy formulation and policy implementation. However, this trend has changed rapidly in recent years with the rise of universal health coverage (UHC); many countries have established new powerhouses for their UHC management. For example, Social Insurance Administration Organization in Indonesia, National Health Agency in India, PhilHealth in the Philippines, and Vietnam Social Security Office in Vietnam.
These institutional changes introduce new challenges for vaccine policy, since the responsibility for vaccine priority setting is often shared by multiple bodies which may have different interests. These challenges are in addition to other contextual and process factors that limit national priority setting for vaccines in low-and middle-income countries (LMICs), including political power, financial constraints, the influence of donor and industry priorities, a lack of explicit frameworks and participation of key stakeholders, and the inappropriate use of priority setting criteria. [1][2][3][4][5] One way of addressing some of the above mentioned challenges is to develop a framework for supporting successful priority setting.
To this end we piloted a new priority setting framework for vaccines, Total System Effectiveness (TSE), which has been renamed to Capacity-led Assessment for Priority-setting of Immunization, in Thailand, where the vertical vaccination program, previously part of the Ministry of Public Health (MOPH), has been reclassified to be part of the newly established and autonomous public health authority called the National Health Security Office (NHSO). The Thai National Expanded Program on Immunization (EPI) was first introduced in 1977. 6 Since then, priority vaccines have been provided free of charge to all the eligible population at public healthcare facilities in Thailand. The EPI currently covers 9 vaccine preventable diseases, namely tuberculosis, hepatitis B, diphtheria, tetanus, pertussis, poliomyelitis, measles, mumps, rubella, Japanese encephalitis, influenza, and cervical cancer. 7 Previously, vaccine prioritization and selection were the joint responsibility of the MOPH and the Advisory Committee on Immunization Practices (ACIP). The ACIP is a subcommittee under the National Vaccine Committee and forms the National Immunization Technical Advisory Group for Thailand. It was established with a mandate to propose priority vaccines to the National Vaccine Committee and formulate recommendations for new vaccine introductions into EPI. The ACIP committee consists of experts in various disciplines including vaccinology, immunology, virology, infectious diseases, epidemiology, public health, and preventive medicine, and is chaired by the Director General of the Department of Disease Control (DDC) under the MOPH. Before 2002, the DDC acted as a vertical program manager in charge of the selection, procurement and management of the EPI program and vaccine management within MOPH. 6 Since the establishment of the UHC scheme in 2002, which currently covers 75% of the Thai population, the NHSO has held the UHC budget independently from the MOPH. 8 It has been responsible for the financial management of all medicines used under the UHC scheme, in addition to all vaccines used for entire population in the country. This has incited a new mechanism for vaccine prioritization and selection and altered the roles of ACIP and DDC/MOPH. The NHSO reference the National List of Essential Medicines (NLEM) as its pharmaceutical benefit package. The NLEM subcommittee, under the National Drug Committee (developed in 1981), is currently the decision-making body for selecting medicines and vaccines for the NLEM. The ACIP has shifted into an advisory role upon the creation of the subcommittee.
Having 2 separate systems for vaccine product prioritization can trigger disagreements on policy decisions regarding vaccine adoption by the UHC manager or its reference, ie, the NLEM subcommittee in Thailand. For example, human papillomavirus vaccine (HPV) was proposed by the ACIP long before its implementation by the NHSO in 2017 and other new vaccines such as Rotavirus vaccine (RVV) and pneumococcal conjugated vaccine (PCV) recommended by ACIP, are yet to be implemented by the NHSO. More detailed information on the ACIP and HPV case study is available in the literature published elsewhere. 7,9,10 Misalignment between the vaccine advisory group and the policy-making body responsible for UHC benefit package development may be inevitable if the 2 adopt different perspectives on vaccine prioritization. Often, the technical advisory group on vaccines focus on safety and efficacy/ effectiveness of vaccine introduction, given the group is mostly comprised of clinicians and are less fixated on the financial and other political pressures in issuing a new vaccination policy. In contrary, the UHC benefit package committee is by nature concerned with implementation and financial sustainability.
Priority-setting processes have been introduced to guide the reimbursement decisions, and health technology assessment (HTA) has been proposed as a priority-setting tool. 1,11,12 HTA is defined as "the systematic evaluation of the properties and effects of a health technology, addressing the direct and intended effects of this technology, as well as its indirect and unintended consequences, and aimed mainly at informing decision-making regarding health technologies. " 13 HTA has proven to help decision-makers improve the efficiency of resource allocation in Thailand. 1,11,12,14 In 2018, the Health Intervention and Technology Assessment Program (HITAP) and the World Health Organization (WHO) undertook a pilot project to understand whether TSE approach could support the harmonization of the vaccine policy process in Thailand. TSE is a concept which primarily aims to strengthen country frameworks for evaluating vaccine products, as well as communicating the priorities expressed by countries to inform regional and global discussions on vaccine preferences. The TSE framework articulates the steps for a structured, systematic and transparent product selection process, based on multi-criteria decision analysis (MCDA). 15 The TSE pilot project was conducted in Indonesia, Mali, Thailand, and Zambia as an exploratory study to assess the feasibility and usefulness of the TSE approach to inform vaccine policy. Thailand was chosen to represent the pilot country with well-established priority setting processes and to compare TSE with such existing mechanisms. Using the MCDA approach, 16 in Thailand's pilot, TSE model was designed as a practical tool run on Excel ® worksheets for supporting the development of immunization policy. TSE allowed local inputs to populate the model and in line with MCDA methodology, it scored vaccine performance against multi decision criteria.
The objectives of this paper are firstly to review the decisionmaking criteria and respective processes of ACIP and the subcommittee of NLEM in vaccine prioritization. Second, to explore whether TSE is a useful approach that can reconcile the 2 current mechanisms in Thailand, by comparing the TSE outcomes with those of ACIP and the subcommittee of NLEM. Lastly, to discuss whether the lessons from "one country, two systems experience" in Thailand can be useful for other countries where the national immunization program are or will become part of the UHC benefit package in the future. The authors will conclude by discussing the potential applications of their findings to other LMIC settings.

Methods
The TSE approach was piloted in Thailand, starting from April to December 2018. For the first objective, authors conducted document reviews of the term of reference for the ACIP and NLEM subcommittee members as well as other publicly available government documents, 17,18 interviews with Secretariat of the ACIP and NLEM (1 person of each Secretariat team), group discussions through key vaccine-related stakeholders meetings for which their minutes are available via this link http://www.globalhitap.net/resources/reportspublications-2/, 19,20 and had direct observation of the meeting between the ACIP and NLEM on the RVV to understand the processes of the ACIP and NLEM subcommittee with regard to vaccine policy decisions. The documents review, the interview with Secretariat of the ACIP and NLEM, and the discussion during the stakeholder meetings were mainly centred on the overall governance, decision-making processes and criteria used for the vaccine prioritization and selection. Open-ended questions were deployed to collect qualitative information and provoke discussions in the stakeholder meeting on the points mentioned above. The discussions were translated into the narrative and figures/tables presented in the results section.
In order to consider the utility of the TSE approach in Thailand, the authors focused on the process-based approach, and used a one-vaccine case study of various RVV products employing TSE and both the ACIP and NLEM subcommittee methodology. Five hypothetical RVV product profiles were developed for the exercise and an excel-based MCDA model was developed specifically for the TSE-based RVV selection in Thailand (TSE pilot), incorporating parts of existing models including UNIVAC, 21 V-TIA, 22 and C4P. 23 This TSE model comprised of 3 main components: decision criteria, estimation of the vaccine performance based on decision criteria, and a scoring and ranking process.
Criteria to be included in the TSE model was decided during the first stakeholder meeting convened in May 2018 with representatives from the NLEM subcommittee and ACIP, relevant ministry departments including DDC, health insurance, academicians, and manufacturers. 19 Stakeholders were asked to complete an open-ended questionnaire stating their most important criteria for selecting a vaccine product in the Thai context and then to subsequently rank the criteria. Due to time constraints, inputs from 15 respondents were collected and the top 5 criteria were used as the decision criteria for the RVV selection model as decided by the stakeholders involved in the TSE pilot study.
Thailand-specific input parameters were computed to make the model relevant locally and to generate the performance scores for the different vaccine products. Parameters inputs included epidemiological data (eg, birth cohort, disease burden and numbers of outpatient and hospitalization), vaccine characteristics (eg, vaccine efficacy and vaccination schedule), cost data and other parameters such as coverage and socio-economic status (Supplementary file 1, Table S1). Parameters were generated from government reports, published literature, and expert opinions. Vaccine characteristics of the 5 hypothetical vaccine products were modified to reflect the Thai context (Supplementary file 1, Table S2). The detailed methodology for estimating the vaccine performance based on decision criteria are shown in Supplementary file 2. 1 USD was considered equivalent to 33.25 baht as per the exchange rates on July 2, 2018 (https:// www.bangkokbank.com/). The threshold for the incremental cost-effectiveness ratio as current used in Thailand was set at 160 000 THB per disability-adjusted life year averted. 24 Scoring was done by giving 100 points to the best performing vaccine and 0 points to the least performing vaccine products. Aggregated scores of the vaccine products were generated by linking the scores on different criteria with the criteria weights. Within MCDA methodology, weights represent the value stakeholders place on the different decision criteria. Given that the pilot study could only reach out to a limited number of stakeholders for criteria selection, equal weights were assigned to all the decision criteria.
In the second stakeholder meeting conducted in August 2018 in Thailand, the TSE findings were presented against the decision-making criteria set by the ACIP and NLEM subcommittee, as well as the TSE stakeholders (as a result of the first meeting). 20

Process of Vaccine Prioritization and Selection in Thailand
The national vaccine policy in Thailand before and after UHC introduction in 2002 is depicted in Figure 1. The ACIP's role remains the same in providing a recommendation on vaccine and immunization policy formation; however, the request needs to be passed to the subcommittee of the NLEM for the inclusion of new vaccines in the reimbursement list. For the vaccines not included in the NLEM vaccines list, individuals must bear the cost of the vaccines themselves.
MCDA has been applied to the process of vaccine prioritization and selection by both the ACIP and the subcommittee of the NLEM. The common criteria considered by both the ACIP and subcommittee of the NLEM are: disease burden, disease prevalence, vaccine safety and efficacy, and estimated budget. In addition to these common criteria, the ACIP also considers local vaccine production capacity while the subcommittee of the NLEM considers cost-effectiveness and equity. For vaccines considered by the NLEM, the ACIP serves as a platform for information exchange, and submits evidence on disease prevalence, diseases burden, safety and efficacy of vaccine to the NLEM subcommittee as inputs for their consideration. The ACIP recommendations typically focus on a list of vaccine types, eg, HPV vaccine, PCV, ranked by their preference order. However, the ACIP do not specify the branding of vaccine products to be included.
The subcommittee of the NLEM, with support from Health Economic Working Group, then produces local evidence on cost-effectiveness and budget impact analysis to feed into the decision-making, 8 ensuring the efficient allocation of resources and financial sustainability for the UHC payer. Like other medicines, NLEM adopts a "choose one policy" which translates as only one specific type of vaccine product/ brand being listed together with their maximum procurement price to inform procurement agencies eg, NHSO or hospitals.

Similarity and Differences in Between TSE and Existing
Processes Used by Both Committees -The ACIP and the Subcommittee of the NLEM Similar to the ACIP and the subcommittee of the NLEM, TSE uses an MCDA approach. However, while the ACIP and NLEM have fixed criteria to consider, the criteria for TSE can be flexible based on the decision question at hand. The output from the quantitative stage of TSE provides a ranking of available options, as is with the ACIP process.
The top 5 criteria identified for the TSE approach, as ranked by stakeholders through the questionnaires, were found to be as follows: safety, health impact, budget impact, delivery costs and cost effectiveness. Safety, health impact and budget impact were ranked by 85%-93% of stakeholders to be among the top 5 criteria (Table 1). Table 2 summarizes the criteria used by the 3 different decision-making bodies.
When assessing the different types of RVV against the base case parameters, no vaccine product performed the best against all 5 criteria. The RVV selection model (TSE-pilot) found that RVV-3 was identified as the top-ranking vaccine, followed by RVV-2 and RVV-4 ( Figure 2). Table 3 shows the quantitative results for the top ranked vaccines using the RVV selection model (results are elaborated in Supplementary file 1, Table S3).
When comparing the results of running the RVV selection test case using the ACIP, NLEM, and TSE approach, all 3 reached similar conclusions (Table 4).
To compare the TSE-pilot, with the existing criteria in Thailand, the scores and rankings of the 5 hypothetical RVVs were evaluated against the ACIP and the subcommittee of the NLEM criteria (Table 5). Thailand's established priority setting mechanisms, the NLEM and ACIP, offered RVV-3 as the optimal choice. In the case of the NLEM process, where only one product can be selected in its "choose one policy, "  RVV-3 would be added to the benefits package. For the ACIP, the results indicate that RVV-3 would be ranked first, RVV-2 would be ranked second and RVV-1, 4, and 5 would be ranked third. As detailed in Table 4, the ACIP criteria ranked the bottom 3 vaccine products with an equivalent score at rank 3, whereas the RVV selection model (TSE-pilot) was able to discriminate between the 3 vaccine products with its scoring and ranking methodology.

Discussion
To our knowledge, this is the first study examining the application of the TSE concept for vaccine introduction in LMICs. The WHO is piloting TSE as an approach for promoting evidence-based, demand driven vaccine selection, in line with existing country policy processes. Given that TSE allows multiple decision-making criteria to be incorporated in a transparent and systematic manner, it may be able to help reconcile concerns currently faced by different decisionmaking bodies in a country with 2 separate public health authorities responsible for policy development and holding technical advisory roles, as is the case with ACIP and NLEM subcommittee in Thailand.
We found that the 3 priority setting approaches, ie, the ACIP's, NLEM's and TSE offered the same policy choice. This may be explained by the fact that all the 3 approaches are based on MCDA with a similar qualitative aspect, use of quantitative evidence and had some overlapping criteria (ie, cost and outcomes/effectiveness data related to vaccine use). Although the top ranked vaccine product may not have performed best on all relevant criteria and is also not the cheapest vaccine option, it emerges as an optimal choice based on the MCDA used by all 3 approaches.
A major difference between the 3 processes is the mechanism by which criteria is selected and included. In the TSE framework, criteria are chosen according to the decision question at hand, whereas in ACIP and subcommittee of NLEM deliberations, the criteria is already articulated. 1,8 Since there are well-established decision-making frameworks and governance in Thailand, such flexibility may not be necessary. However, when considering TSE at the global level, this flexibility to select the criteria based on decision question can allow for better discrimination between options. This would encourage countries to identify the decision criteria that is locally relevant, and may prompt the consideration of a broader set of criteria than the scientific considerations  commonly considered by National Immunization Technical Advisory Groups. Differences in perspectives and definitions of the criteria adopted in the 3 approaches were observed. For example, the ACIP includes disease burden in the way of measuring health impact and measuring case fatality rate whereas the RVV selection model (TSE-pilot) captures rotavirus cases averted as an indicator to account for the health impact. In addition, the application of the ACIP and NLEM criteria to different products of the same vaccine may not be able to differentiate if based on the epidemiological criteria eg, disease prevalence and burden. TSE can be seen as having an advantage over the ACIP mechanism as it is able to make a stronger differentiation between products of the same vaccine type. TSE can be designed to include a wide range of criteria for a one-vaccine evaluation. Whereas the ACIP mechanism has been established with the purpose of evaluating across vaccine types, with a strong focus on clinical aspects. Thus, in this study, while TSE could identify differences in each products performance in preventing rotavirus cases, ACIP ranked all 5 products the same in terms of disease burden and prevalence.
During the first stakeholder meeting, the chair of the  subcommittee of the NLEM expressed the importance of decision-making bodies on health intervention and technology adoption being independent and policy neutral. 19 They highlighted the value in the ACIP and the subcommittee of the NLEM fulfilling different functions as it enables the 2 committees to hold each other account. In the current status quo, the ACIP make an initial shortlist based predominantly on public health benefit, then the subcommittee of NLEM considers whether there is an economic argument for adding the recommended vaccine to the benefits package, in view of health interventions more broadly. Thus creating a checkbalance mechanism that ensures that the introduction of a new and high-cost vaccine to the national reimbursement list is cost-effective, sustainable and affordable for the country. Nonetheless, the exercise of utilizing the TSE framework jointly with ACIP and NLEM could encourage a platform for deliberation in terms of the stakeholders, criteria, and the evidence involved in each process and could bring about better alignment between the 2. This may be true for other countries in which the immunization and broader UHC decision-making processes or financing streams are not combined. TSE may thus be able to help reconcile different interests and concerns regarding vaccine introduction policy between different priority setting agencies in UHC countries. This study highlighted that, if used in real policy, the TSE approach may bring about a significant shift in the political and economic aspects of vaccine decision-making in Thailand. Given that TSE requires technical expertise in economics and modelling, individual with these skills set eg, health economists, analysts, technocrats would need to play a more active role in advising policy than in the current system at the ACIP, which focuses predominantly on the clinical aspects of the vaccine. Although the NLEM applies health economic evidence alongside clinical information, there is a deliberative process that allows qualitative data (such as urgency and equity concerns) to be included in the decisionmaking process. The quantitative TSE model was more rigid in this sense, though it can make vaccine decisions more transparent and consistent. There are some limitations in the study design. While the RVV selection model developed to reflect the TSE approach provided the flexibility to incorporate country priority issues and generate good quality quantitative evidence, it required locally relevant specific inputs (parameters). Precision decisions depends on an accuracy and quality of data which can be difficult for other LMIC settings faced with data challenges. Similarly, implementation of the model requires technical capacity to be able to perform the analysis. These challenges can be more evident once the 4 country pilots have been carried out. In addition, the model focused on quantifiable criteria that can be measured in a numerical format. As a result, some policy concerns regarding the introduction of new vaccines such as equitable access or domestic vaccine production capacity, cannot be easily incorporated in a quantitative model as shown in the case study. Lastly, inputs from a limited number of respondents were included and only the top 5 decision-making criteria was collected, for example representatives from the public and/or civil society were absent from the first meeting. However, this is an exploratory study and to this end the study was not designed to reach all relevant stakeholders in the Thai healthcare system for eliciting the decision criteria. The analysis applied equal weights across the TSE criteria for simplicity due to the non-representativeness of the stakeholder participation in the meeting. The readers should keep in mind the potential impact of the above limitations on the study findings.

Conclusion
Findings from the Thailand pilot indicate that TSE may be a beneficial approach for LMICs that have not yet developed strong and accountable decision-making processes and governance structures for immunization policy. TSE has the potential to provide a well-reasoned and systematic approach for these countries, as a starting block for structured and transparent decision-making for vaccine product selection. Furthermore, TSE could be of value to a broader set of countries, including Thailand, for aligning prioritization mechanisms and thus as a foundation to help streamline vaccine policy processes.