How Are New Vaccines Prioritized in Low-Income Countries? A Case Study of Human Papilloma Virus Vaccine and Pneumococcal Conjugate Vaccine in Uganda

Background: To date, research on priority-setting for new vaccines has not adequately explored the influence of the global, national and sub-national levels of decision-making or contextual issues such as political pressure and stakeholder influence and power. Using Kapiriri and Martin’s conceptual framework, this paper evaluates priority setting for new vaccines in Uganda at national and sub-national levels, and considers how global priorities can influence country priorities. This study focuses on 2 specific vaccines, the human papilloma virus (HPV) vaccine and the pneumococcal conjugate vaccine (PCV). Methods: This was a qualitative study that involved reviewing relevant Ugandan policy documents and media reports, as well as 54 key informant interviews at the global level and national and sub-national levels in Uganda. Kapiriri and Martin’s conceptual framework was used to evaluate the prioritization process. Results: Priority setting for PCV and HPV was conducted by the Ministry of Health (MoH), which is considered to be a legitimate institution. While respondents described the priority setting process for PCV process as transparent, participatory, and guided by explicit relevant criteria and evidence, the prioritization of HPV was thought to have been less transparent and less participatory. Respondents reported that neither process was based on an explicit priority setting framework nor did it involve adequate representation from the districts (program implementers) or publicity. The priority setting process for both PCV and HPV was negatively affected by the larger political and economic context, which contributed to weak institutional capacity as well as power imbalances between development assistance partners and the MoH. Conclusion: Priority setting in Uganda would be improved by strengthening institutional capacity and leadership and ensuring a transparent and participatory processes in which key stakeholders such as program implementers (the districts) and beneficiaries (the public) are involved. Kapiriri and Martin’s framework has the potential to guide priority setting evaluation efforts, however, evaluation should be built into the priority setting process a priori such that information on priority setting is gathered throughout the implementation cycle.


Background
Vaccines are considered to be cost-effective health interventions, and play an important role in reducing under 5 mortality if high coverage is achieved. 1 Building momentum from the Millennium Development Goals (MDGs), particularly those focusing on children, development assistance partners, such as the World Health Organization (WHO), United Nations International Children's Emergency Fund (UNICEF), and Global Alliance for Vaccines and Immunization (GAVI) have become increasingly interested in and committed to facilitating the development of new vaccines and scaling up the implementation of existing vaccines. For example, in 2010, the Gates Foundation committed $10 million to vaccine research and the development and delivery of vaccines to the world's poorest populations. 2 Growing interest in vaccines as a useful health intervention has led to a "vaccine boom. " By the end of 2008, there were more than 120 vaccine related products developed -the highest in a century. However, not all vaccines that are developed are recommended for adoption. Since 2010, the WHO has recommended at least 6 new vaccines for adoption by low-income countries (LICs), these include pneumococcal conjugate (PCV), dengue, hepatitis A, influenza, rotavirus, and human papilloma virus (HPV) vaccines. 2 Furthermore, there are about 25 vaccines in the research and development pipeline. 1 These new vaccines, if adopted, have to compete for the meager public health resources within LICs, who are already responsible for the provision of at least 6 existing routine vaccines. Financial and human resources, as well as physical infrastructure, are necessary for the successful implementation of any vaccine program. The resource requirements for adopting new vaccines are greater than for existing vaccines. For instance, some new vaccines have protocols and cold chain requirements that are different from the routine vaccines and require retraining health workers and re-equipping health units. 3 Prioritizing between new vaccines, while at the same time sustaining the existing vaccine program, is a major challenge for governments and Ministries of Health in LICs. Vaccine prioritization should occur through a clear and explicit process, as decisions to select one vaccine may occur at the cost of not investing in another and have consequences for health equity and outcomes. Studies have described the decision-making processes involved in adopting new vaccines in LICs. For example, Burchett and colleagues conducted a study on the decision-making process for introducing vaccines in 12 LICs and identified 11 criteria that influence the adoption of new vaccines. 4 Expanding on these issues, Mantel and Wang discuss the adoption of new vaccines in a commentary. 5 While these studies provides insight into the prioritization of new vaccines, Burchett and colleagues identify a need for more detailed case studies that expand our understanding of the process. In particular, they note that more research is needed to explore larger contextual issues such as political pressure and stakeholder influence and power, and whether the implementation of vaccine priorities reflects official national-level objectives and processes. 4 This information is needed to identify the most effective ways to improve decision-making, and ultimately, health outcomes. This paper provides an in-depth exploration of priority setting for new vaccines in Uganda. The national and sub-national decision-making dynamics in the Ugandan public health system are similar to those in other LICs, and thus, provide a basis for generalizations beyond the Ugandan context. Based on the critical role that donors play in the adoption of new vaccines, 3 this paper also considers how global priorities can influence country priorities. This study evaluates priority setting for vaccines in Uganda using Kapiriri and Martin's conceptual framework for evaluating priority setting. 6 This analysis is based on two specific vaccines, the HPV vaccine and the PCV.

Methods
The Analytical Framework Kapiriri and Martin's framework was developed in 2010 and is based on the priority setting literature and Delphi interviews with researchers involved in health prioritization research in low-and middle-income countries (LMICs). In 2015, the framework was validated by researchers and policy-makers in both global and LMIC contexts (Forthcoming), making it particularly relevant for an examination of priority setting for new vaccines in Uganda. The framework is holistic since it addresses the potential roles played by the social, cultural, political and economic context, which cannot be overlooked when discussing priority setting in LMICs. Kapiriri and Martin's framework identifies 23 major parameters for successful priority setting (see Table 1). 6 Each parameter is organized according to whether they are (1) internal/external relative to the priority setting institution, and (2) immediate/delayed; whereby immediate measures are expected within a planning/fiscal/budget year and delayed measures are expected beyond 3 fiscal years. Each parameter is associated with objectively verifiable indicator(s) (OVI) and means of verification (MOV). These specify the evidence that will inform evaluators if an expected measure has been achieved. By being objectively verifiable, these indicators permit different people, using the same measuring process, to obtain the same results independently. 7 Data Collection This is a qualitative study involving a review of documents and key informant interviews. Data collection strategies were aligned with the MOV proposed in the framework, where possible. Document Reviews Policy documents and media reports were reviewed. Policy documents include the health sector strategic and investment plans (HSSIPs) and the Uganda National Expanded Program on Immunization's (UNEPI's) comprehensive multiyear plans for immunization (CMYPs) between 2000-2015. [8][9][10][11][12][13] While the HSSIPs describe the country's priorities for new vaccine adoption, the CMYPs show what is actually implemented. Media reports on HPV and PCV published within the same period were also reviewed. Media reports provide details of the rationale for the priorities set, and the process of rolling out vaccines to the public.

Implications for policy makers •
An increasing number of new vaccines on the market and limited resources will always necessitate policy-makers to make informed decisions about which vaccines to introduce first. • Using the examples of the prioritization of human papilloma virus (HPV) and the pneumococcal conjugate vaccine (PCV), Kapiriri and Martin's framework was able to identify areas of good practice and areas that require improvement. • There is a need to integrate real-time evaluation into the priority setting processes for new vaccines such that information is gathered throughout the implementation cycle rather than ex-ante.

Implications for the public
New vaccines have the potential to improve quality of life, especially for those living in low-income countries (LICs). However, the number of new vaccines that LICs can potentially adopt is often greater than what LICs and their development assistance partners can afford to introduce. For this reason, LICs need to follow a priority setting process when identifying which vaccines to introduce first. Recommendations to improve priority setting for new vaccines should be based on an understanding of how these decisions are currently made. This study evaluates the decision-making processes that led to the introduction of two new vaccines -PCV and HPV -from the global level to the point of delivery in Uganda. While the decision-making process was appropriate in some areas, it also fell short in others; for instance, there was a lack of publicity about the need to set priorities and the rationales behind the priorities that were identified. Transparency requires that decisions and their rationales are available to the public. Media reports also showcase public awareness of the priority setting process; the public is a key stakeholder in vaccine priority setting and new vaccine introduction requires intensive communication and social mobilization targeting communities to provide information about new vaccines.

Interviews
Interviews were conducted between 2013 and 2015.
Respondents were identified from the 3 main levels of priority setting -the global, national and sub-national (district levels). Fifty-four interviews were conducted in total (see Table 2). Global level respondents provided context for understanding how global level priority setting impacts national level priority setting. National level respondents described the priority setting process within Uganda. Follow-up interviews with national actors provided information on which priorities were implemented and why. District level participants were critical since, according to the national health policy, district managers are responsible for policy implementation in Uganda. These participants provided insight into the enablers and barriers to policy implementation.

Study Sample and Sampling Strategy
Purposive sampling was used, whereby only people deemed to be knowledgeable of, or involved in, new vaccine introduction were recruited. Respondents at the global level were identified through their organization's web page. Within Uganda, the initial respondent was the officer in charge of the Expanded Program on Immunization (EPI); he identified subsequent respondents. Three districts were selected to reflect diverse settings, (ie, regional -North, Central and East, rural/urban, and new/old districts). The interviews were conducted by the principal investigator (LK) and trained Ugandan research assistants, using a pilot-tested interview guide. Global level interviews were conducted by telephone while the Ministry of Health (MoH) and district interviews were conducted face to face. Interviews lasted an average of 45 minutes and were audio recorded with permission from the respondents.

Data Analysis
Interviews were transcribed verbatim and data were analyzed using Nvivo 10. Initially, 3 members of the research team independently read and coded two interviews, developing a list of codes. They then convened to discuss the code names and any discrepancies and to develop an agreed upon code list. Similar codes were grouped into categories and related categories linked to overarching themes. Further analysis involved using Kapiriri and Martin's framework 6 to assess the degree to which the described process met the parameters for successful priority setting (summarized in Table 1). 6

Results
Documented Priority Setting Processes for New Vaccines at the Global Level World Health Organization According to respondents, vaccines prioritized at the national level are chosen from a global menu of approved vaccines. This menu is created from a list of recommendations put forward by the WHO. Once a new vaccine product is developed and presented to the WHO, WHO seeks advice from its Strategic Advisory Group of Experts (SAGE). SAGE convenes advisory working groups with the relevant expertise to synthesize evidence and produce a position paper on new vaccines, which is publicized at the global, regional and national levels. SAGE provides recommendations to WHO, who present their recommendations to the World Health Assembly. This information filters down to the WHO in country and regional offices, who encourage countries to adopt the new vaccines. The criteria used by WHO for vaccine prioritization include: demand in the respective UN-supplied markets, WHO compliance needs with international health regulations, eradication, elimination or control initiatives and immunization program considerations, and finally, SAGE recommendations and the security/sustainability of supplies. 14

Global Vaccine Alliance
While new vaccines may be recommended by WHO, the board of the GAVI independently approves the menu of vaccines that will be subsidized. WHO and GAVI priority vaccines are summarized in Table 3. GAVI undertakes a very systematic and rigorous process called a vaccine investment strategy every 2 to 3 years to determine which vaccines to support. Two of the most important criteria that GAVI uses to determine which vaccines to prioritize are cost-effectiveness and burden of disease. Once the menu has been finalized, GAVI calls for proposals from countries that are eligible for GAVI subsidization. Only LICs with a Gross National Product of less than $1580 are supported by GAVI. 15 GAVI eligible countries must submit their proposals for vaccine introduction through country multi-year plans. After the submission of the proposal to GAVI, an assessment of the in-country capacity of the cold chain system and other logistics, known as an Effective Vaccine Management Assessment (EVMA) is conducted. This is followed by an Effective Vaccine Management Improvement Plan to improve the health system's capacity to manage the new vaccine. This evidence is fed into the country's GAVI proposals to indicate the country's ability to sustain the introduction of the vaccine. GAVI's new vaccine introduction support consists of providing eligible countries with a substantially subsidized supply of the vaccine, as well as a one-time grant to assist with the initial expenses of vaccine introduction. The grant does not cover the full costs of new vaccine introduction, but rather works ''to facilitate the timely and effective implementation of critical activities in the national vaccine introduction plan in advance of a new vaccine introduction and cover a share of the pre-introduction activities. " 15 Multiple respondents described how GAVI affects the prioritization process at both the global and national levels, by influencing the choice of vaccines included on the new vaccine menu and through directly funding specific vaccines. "…the country is not allowed to get other vaccines partly because of the GAVI contract…" [G_8]. "...So yea if GAVI puts for us a menu of vaccines that they're subsidizing it also kind of affects the priority setting process... " [G_4]. GAVI approves the menu of vaccines that can be subsidized in LICs. However, vaccines adopted are approved through extensive and participatory national level processes. These national level processes are described below.
Documented Priority Setting Processes for New Vaccines at the National Level According to respondents, priority setting for new vaccines in Uganda is part of the national level priority setting process which involves the development of the HSSIPs, where 5-year national MoH priorities are identified by a broad range of stakeholders including government officers, development assistance partners, civil society organizations, and representatives from the districts. According to these reports, immunization and introduction of new vaccines have been consistently identified among the national priorities. 8-10 The 5-year plans specifically identify the new vaccines that should be introduced during the 5-year period. These decisions are guided by evidence presented by the medical officer in charge of the EPI, which has been derived from prior processes that are focused specifically on prioritization of new vaccines (described below). National level priorities are then presented to the Health Interagency Policy Advisory Committee (HIPAC), which gives the final approval and monitors vaccine implementation.

Processes for New Vaccine Prioritization
The prioritization process for new vaccines involves different institutions and stakeholders namely: the Uganda National Academy of Science (UNAS), Uganda National Immunization Technical Advisory Group (UNITAG), or the Advisory Committee on Vaccines and Immunization (ACVI), the National EPI, the Maternal and Child Health Technical Working Group (TWG) and the HIPAC. The process is often triggered when the WHO and/or GAVI recommends a new vaccine. UNAS then takes the responsibility of electing a relevant advisory committee for the new vaccine, referred to as the NITAG or the ACVI. The composition and roles of these different institutions and committees are summarized in Table 4. The UNAS has developed clear criteria for identifying members of the NITAG and explicit guidelines that the NITAG should use when evaluating the introduction of new vaccines. The NITAG collects and reviews evidence on the potential new vaccines and makes evidence-based recommendations to the UNAS. Based on these recommendations, the UNAS advises the TWG within the MoH, (chaired by the EPI manager). Once the TWG makes the decision to recommend the introduction of a new vaccine, the EPI manager develops a proposal and presents it to the senior management committee in the MoH; once approved this proposal is submitted to HIPAC. If Uganda were able to fund the introduction of the new vaccines, the next step would have been for the national medical stores to purchase the approved vaccines. However, since the country cannot afford the costs of new vaccines, they have to apply for funding from GAVI. Therefore, after the government structures have approved the introduction of a new vaccine -a critical condition for GAVI funding -the EPI manager includes those new vaccines in the proposal for GAVI funding. If a new vaccine is approved by GAVI, the country is funded to introduce the vaccine. However, before the funding is approved, GAVI conducts an in country EVMA, to establish the country's readiness to introduce the new vaccine. Next, in country stakeholders utilize feedback from the GAVI secretariat to prepare an effective vaccine management improvement plan to ensure that the country is ready to implement the new vaccine. This may involve establishing in country surveillance to establish baseline indicators at introduction, cold chain preparation, waste management preparation, editing data management forms, health worker training, communication and community mobilisation. Once the recommendations have been implemented, GAVI releases the funds to facilitate the introduction of the new vaccine. Vaccines are then procured at the national level and delivered to the districts. While the districts are partly in charge of implementation, the EPI and the MoH are in charge of disease surveillance and reporting of adverse effects. The existing process has been described above. However, this process is not always followed, as was exemplified in the cases of introducing PCV and HPV.  Table 5). It was not possible to evaluate some of the parameters namely: efficiency of the priority setting process, reflection of public values, increased public awareness of priority setting, increased public confidence in and acceptance of decisions, and decreased resource wastage.

Allocation of Resources According to Priorities
According to this parameter, resource allocation should be based on the priorities that are identified. While the decision to introduce PCV was deemed appropriate by our respondents, several thought that the allocation of resources to HPV in 2005-2010 and 2010-2015 in the CMYPs was inappropriate since this did not match the new vaccines that were prioritized in the HSSIPs. For instance, in HSSIP III, 8 the focus was on yellow fever and hepatitis B, yet resources were allocated to the introduction of HPV, in addition to hepatitis B and rotavirus. 12

Stakeholder Participation
This parameter relates to whether relevant stakeholders were involved in the priority setting process and were able to meaningfully contribute to the process. While the documents described a participatory process and explicit participation structures, our interviews revealed that not all the relevant stakeholders were actually involved in the process and that some stakeholders were involved in an inappropriate way. Stakeholders who were not adequately involved in the priority setting process include: MoH officers, district officers and the public. According to respondents, development assistance partners were involved in the priority setting process an inappropriate way. The participation of these stakeholders is discussed in detail below. . There was a sense that development assistance partners' adherence to a consultative priority setting process had decreased. Study participants who are members of development assistance partners argued that development assistance partners dominated forums for harmonizing partners, such as the Immunization Inter-Agency Coordinating Committee (IACC). According to one respondent, contrary to its intended role, the IACC had become "an opportunity for each partner to come with his influence" and to "push in a specific direction" and "the government has very little say in this type of forum" [G_7].

Ministry of Health Officers
While participants reported that the involvement of development assistance partners were uneven, they also alluded to 2 special categories of stakeholders -MoH officers and districts -who they thought were critical to the implementation of the new vaccine program, but were insufficiently involved in setting priorities for vaccines. There were sentiments that there was limited involvement of the MoH officers in the development of HSSIP III since at the

District Officers
Respondents reported a disconnect between national initiatives, such as the introduction of new vaccines, and processes within the districts. Participants argued that in many cases the districts are neither consulted during the priority setting process, nor made aware of the national vaccine priorities. This results in the districts failing to implement the vaccines since they have not put strategies in place to operationalize them. For example, 2 participants argued:

"…Currently I think the link [between the national and district priority setting] is very poor"
[U_2].

"[The districts] didn't have an annual work plan. So I don't know what they're implementing. So implementation is not being guided by priorities in the district" [U_10]
. District-level respondents argued that they had not been given adequate opportunities to participate in priority setting discussions at the national level; they suggested that this was problematic since only individuals working at the district level could truly understand the specific needs of their population, and the feasibility of proposed interventions.
"We need to set our own priorities according to the local context and local needs. We would be doing very well if we did...So the issue of decentralization is also needed to be looked at, it needs to be considered, and the district be empowered more so that they are able to implement their priorities" [D_3].

The Public
When setting priorities, it is important that the public, as the beneficiaries of the process, are involved. This can be achieved through having public representation on the priority setting committees or in institutions or by conducting public surveys where public opinions are considered during the priority setting process. The documents reviewed showed that there are efforts to include representatives from the public on various committees, such as the NITAG, that set priorities for new vaccines. The public is also represented through civil society organizations that participate in the HSSIP processes. This study was unable to assess actual participation of the public at these fora. In addition, the document review shows that, community level studies were conducted before the introduction of HPV and PCV. However their purpose was to assess the public's readiness and their perceptions about the vaccines. 16

Use of Evidence Pneumococcal Conjugate Vaccine
According to Kapiriri and Martin's framework, 6 using evidence to guide priority setting can discourage the use of controversial criteria, such as politics, hence increasing the appropriateness of decision-making. The review of GAVI applications also showed that the MoH placed considerable emphasis on generating locally-relevant data to support the introduction of the PCV vaccine. For example, a sentinel disease surveillance and reporting system for pneumococcal disease was initiated in 2002, which generated evidence that supported the case for the introduction of PCV. Evidence on the high disease burden of invasive pneumococcal diseases such as meningitis and pneumonia (respectively 28/1000 and 3212/1000 for children below 5 years), the anticipated benefits, especially for children, and the health system's readiness, as demonstrated by the EVMA, all supported the introduction of PCV. 13 In addition to this, several studies were conducted prior to the introduction of PCV, for example, a cost-benefit analysis study 18 and another study exploring the role of the public. 19 The evidence from the Ugandan surveillance system supported the selection of PCV-13, which addresses more strains of pneumococcus and is more compatible with the Ugandan context, in terms of cost, ease of handling, management, storage, and fit in the immunization schedule; however, excessive demand for PCV-13 at the global level led to the MoH being forced to choose PCV-10 instead of the more locally appropriate PCV-13.

Human Papilloma Virus Vaccine
Several studies were conducted and published prior to the introduction of HPV vaccine in Uganda. Some of these studies examined opportunities and obstacles in delivering the vaccine 17,20 formative research by PATH 21 and research on delivery and readiness. 16,22 However, it is not clear if these studies were conducted in preparation for the implementation of the HPV vaccine. They may have not influenced the initial prioritization of the vaccine.

Use of Explicit Relevant Priority Setting Criteria
In order for priority setting to be fair, it should be based on relevant reasons or information such as epidemiological information, considerations of equity, or data on costs and effectiveness of interventions. Discussions with participants and the HSSIPs indicate that acceptable criteria, such as use of evidence, and considerations of equity, in addition to unacceptable criteria, such as political sentiment, and industry and donor priorities influenced the decision to prioritize the HPV and PCV vaccines.

Pneumococcal Conjugate Vaccine
Our interviews with study participants and a review of documents reveals that considerations of equity, (since PCV focuses on children who are considered to be vulnerable), cost effectiveness, and burden of disease were critical in the MoH's choice to introduce PCV. For example, one respondent explained that based on cost effectiveness, PCV was planned to be introduced ahead of vaccines such as HPV.

Human Papilloma Virus Vaccine
Unlike PCV, there is limited information on the criteria that facilitated the MoH's prioritization of the HPV vaccine. Participants described how equity was an important criterion considered in the prioritization of HPV. According to one Ugandan policymaker, the HPV vaccine targets women, and its prioritization was a result of the importance of gender equity. He explained that the MoH's prioritization of the HPV vaccine is a good example of the way in which equity as a criterion can trump other considerations such as evidence of burden of disease: "I think HPV was likely taken on for equity reasons…Women felt they were being ignored…if we were to set priorities based on morbidity patterns [alone], I think HPV wouldn't have appeared" [U_ 9]. Some participants identified criteria that they thought not to be relevant to prioritizing new vaccines. For example, global participants argued that HPV became a priority because pharmaceutical companies promoted the vaccine, while national respondents thought it was partly influenced by political sentiments. According to one participant, industries are more likely to promote the introduction of more profitable vaccines like HPV compared to vaccines for diseases such as measles: "…if there is a donor with a lot of money and he has a new product… they will find a way.

Publicity of Priorities and Criteria
Transparency is important to fair decision-making. The publicity condition requires that the decisions and the rationales and criteria on which they are based are available to the public. In Uganda, attempts to publicize decisions occur mainly through the print media. There were a couple of media reports on the introduction of the new vaccines, with varied content. For example, newspapers reported on the incidence of meningitis and cervical cancer, the government's plans, with support from GAVI, to introduce PCV, 23 and the HPV pilot implementation. 21 Furthermore, there were media reports about the magnitude of the health problems being addressed and the need for the public to respond positively. 24- 26 The launches of vaccines in many districts (for example, the launch of PCV by the president) were also often highly publicized. 23 Other media reports were about the limitations of the new vaccine programs with some questioning the sustainability of the funding (given the prior misappropriation of funds), and history of vaccine stock outs. [29][30][31][32] In some of the newspaper reports, the public reacted negatively to HPV. Although the decision to adopt both vaccines was disseminated to the public through popular media, these media reports did not inform the public of the process and rationale used to identify the vaccine priorities.

Functional Mechanisms for Appealing and Revising the Decisions
According to this condition, there should be mechanisms for appealing the decisions and opportunities for revising the decisions, based on new evidence. This study did not identify any explicit appeals and revisions mechanisms. The only avenue for channeling complaints seems to be through the media (discussed above).

Functional Mechanisms for Enforcement
According to Kapiriri and Martin's framework, 6 there should be internal or external mechanisms to ensure a fair process. While the MoH does have structures that oversee the implementation of the new vaccines, there were no mechanisms that explicitly focused on assuring that the 4 conditions of a fair process are met.

Fairer Priority Setting Processes
According to the framework, priority setting should be perceived as fair. A fair process meets the following conditions: relevance -ie, stakeholder participation, use of evidence, use of explicit relevant criteria -publicity, revisions and enforcement (see Table 4). 33 From the above discussion it is clear that while there were attempts to meet the conditions of a fair process, none was fully met.

Increased Public Confidence in and Acceptance of Decisions
As discussed above, newspaper reports suggested that the public reacted negatively to the introduction of the HPV vaccine. However, no explicit complaints or revisions to the decisions were documented or mentioned by respondents.

Increased Stakeholder Understanding, Satisfaction and Compliance With the Priority Setting Process and Decreased Dissentions
Stakeholders are key to the success of any priority setting process. Stakeholder involvement in the process or their exposure to the publicized information (discussed above) would facilitate their understanding of priority setting. Their involvement would also contribute to their satisfaction and compliance with the outcome of the process and would result in reduced dissentions with the decisions. While stakeholders seemed satisfied with the documented "ideal" process, there was dissatisfaction with the actual process, especially in relationship to the prioritization of HPV in 2005-2010 and 2010-2015 in the CMYPs since the actual process did not match the HSSIP. Informants' comments suggest that the process of setting and implementing priorities was not inclusive, and that there was decreased understanding and satisfaction with the process, which resulted in apathy in implementing priorities, especially within the districts. While stakeholders reported dissatisfaction, no formal complaints were reported.

Improved Internal Accountability/Reduced Corruption
Accountability ensures that stakeholders gain confidence in the priority setting process and the priority setting institution. This confidence will increase stakeholders' compliance and hence facilitate more appropriate use of resources, augmenting successful priority setting. Interviews with stakeholders from the MoH revealed that although stakeholders were aware of the institutions' existing priority-setting process, there was limited transparency and understanding of the actual prioritization process in the case of HPV.

Strengthening of the Priority Setting Institution and Impact on Institutional Goals and Objectives
In some instances the priority setting institution may be independent of the MoH, however, in the case of priority setting for vaccines, the MoH is responsible; therefore, the parameters related to the priority setting institution are assessed under the health system parameters.

Impact on Health Policy and Practice
The priority setting process for the new vaccines did not impact health policy. However, the implementation of the new vaccines did change health practice since at the time of the study, PCV had already been approved for integration into the routine vaccination schedule and there were plans to do the same for HPV.

Achievement of Health System Goals
Successful priority setting should have a positive impact on the health system's achievement of its goals, which include improving population health. The effectiveness of PCV and HPV vaccines is well documented. While it was not possible to objectively assess the impact that the vaccines have had on alleviating the disease burden, the population coverage of the vaccines can be used an indicator for their potential impact.
At the time of the study however, HPV had not been rolled out in the whole country.

Improved Financial and Political Accountability
Financial accountability relates to improved tracking and reporting on financial allocations, disbursement and utilization of financial resources, while political accountability relates to the degree of integration of public interests. One respondent from the MoH argued that donor influence, combined with weak leadership and lack of financial accountability on the part of those working in the MoH had resulted in a situation in which most of the resources for immunization are externally controlled, which impacts timely disbursement and implementation. . The above sentiments were also alluded to in the newspaper reviews where delays in district accountabilities for GAVI funds for PCV introduction reportedly resulted in delays in disbursements of other funds. This led to delays in key activities necessary to prepare the health system for PCV introduction, particularly health worker training. In turn, there were delays in the final roll out of the vaccine, even when it had already been launched. 32

Increased Investment in the Health Sector and Strengthening of the Healthcare System
For the organizations that invest in health systems in LICs, increased financial accountability often builds their trust and increases their investments into the health system. Furthermore, a successful priority setting process should facilitate the strengthening of the priority setting institutions. In turn, strong institutions are likely to implement priority setting successfully. According to the interviewees, the priority setting process resulted in increased investment in the health sector through the introduction of GAVI funding to support the MoH's delivery of HPV and PCV. The introduction of the new vaccines resulted in the training of health personnel, strengthening the cold chain and strengthening the disease surveillance system. However, these resources and activities were directed narrowly towards vaccines. GAVI funding to support the MoH's delivery of HPV and PCV increased investment in the health sector. However, respondents argued that development assistance partners' influence on the prioritization of new vaccines was problematic, because of the hidden costs to national health systems associated with the so-called free vaccines. They emphasized the considerable work and resources from the health system required to introduce new vaccines such as HPV; in addition to co-financing, participants discussed additional costs, such as the cold chain infrastructure and training for health workers required. For example, a participant argued that in the past, the influence of development assistance partners was less problematic since vaccines were always a "good buy. " However, newer vaccines such as HPV are more costly. This means that it is important for development assistance partners to support countries to make decisions based on national preferences and good data. "

Contextual Factors
Although the MoH had introduced the 2 vaccines, participants identified contextual issues that were reported to have weakened the institutional capacity of the MoH. These negatively impacted the priority setting process for both vaccines.
Participants identified a number of institutional and resourcerelated weaknesses in the health system, which impacted institutional capacity and continuity, limiting the MoH's ability to implement the stated priorities. Respondents from the MoH described how due to apathy and a lack of leadership, priorities stated in the HSSIPs failed to be implemented. A major issue discussed by participants is the negative influence that changes in staffing and leadership at the MoH had on the implementation of HSSIP II and the design of HSSIP III. During the implementation of the most recent HSSIPs, there were multiple changes involving the Ministers of Health and their technical arm, the Director General. New appointments reportedly resulted in poor transitional processes. This inevitably interrupted the implementation of identified priorities. Furthermore, respondents reported that some of the new leaders seemed anxious to introduce new initiatives, while others were not knowledgeable about the previous HSSIP.
Respondents also discussed the negative impact of a lack of resources on the institutional capacity for the implementation of vaccine priorities. Participants described 2 seemingly contradictory issues in relationship to the lack of resources at the national level: a lack of resources, and the MoH's inability to use existing resources wisely. While resources may be available at the national level, district respondents reported that the disbursement of funds from the Ministry to districts for the implementation of priorities was often delayed considerably, which negatively affected the implementation of priorities. With regard to the latter, one district-level Another major challenge discussed by participants is a stagnant budget for vaccines. Respondents described how the budget for routine vaccines has not grown, however, the number and cost of new vaccines such as HPV and PCV is expanding, in addition to the population size, number of districts and the disease burden, which increases the pressure on the national budget. As one participant noted: "…But overall the budget for the health sector over the last few years has actually not grown…actually it has been going down and yet we have seen population but also disease burden…not just burden but the case mix of diseases has been expanding but also …the packages have been expanding but not necessarily the resources" [U_18]. To ameliorate these issues, global level respondents suggested that development assistance partners should align their priorities with those of national governments. However, as one technical partner argued, while the concept of donor harmonization has been promoted, it has not been well operationalized; the harmonization of the stakeholders involved in vaccine priority setting is negatively impacted by weak institutional capacity in countries such as Uganda: "…harmonization means that everybody agrees that we need to do things differently. Partners should align among themselves in the first place and secondly they should align with countries' priority. But it is still a challenge to…to operationalize this approach at country level and I think it is because you know the system in most of these countries is still weak…So each partner will then go, you know, their own way because it is just difficult for the national health system to coordinate all this. And also the partners want quick results and if there isn't results coming very quickly then they will do the things by themselves. So the way I think, what will help to…really sort out this is to have strong system and institutions in the countries…" [G_6].

Discussion
This paper contributes to the current literature by describing and evaluating priority setting for 2 new vaccines in Uganda using Kapiriri and Martin's framework, 6 which was developed to evaluate priority setting in these contexts. To our knowledge, this is one of the first case studies to examine the influence of the global, national and sub-national levels on priority setting for new vaccines. While this study draws on the Ugandan context, the national and local decision-making dynamics in the Ugandan public health system, as well as the economic and political power differences between its MoH and development assistance partners, are similar to those in other LICs, and thus, provide a basis for generalizations beyond the Ugandan context. The parameters of successful priority setting are summarized in Table 5. First, the parameters that could be assessed and whether or not the priority setting processes satisfied these quality indicators is discussed. Lastly, the parameters that could not be assessed and the implications of these findings are discussed (see Table 4). Based on the parameters in Kapiriri and Martin's framework, 6 the process for prioritizing both vaccines was successful in a few ways. First, although no framework was used in the priority setting process for either vaccine, our data suggest that the decisions to introduce both HPV and PCV were based on considerations of evidence; while HPV used evidence from a pilot project in 2 districts, PCV used evidence from sentinel surveillance reports and community studies. In addition, both vaccines were prioritized based on explicit relevant criteria: cost-effectiveness for PCV and burden and disease and equity for HPV. While stakeholders from the MoH were generally satisfied with the decision to introduce PCV, several expressed dissatisfaction with the prioritization of HPV in 2005-2010 and 2010-2015 in the CMYPs since the actual process did not match the HSSIP. Stakeholders' understanding of the priority setting process for PCV was better as compared to HPV since PCV was one of the original priorities set by the MoH, while HPV was not. In the case of HPV, the allocation of resources toward the vaccine did not successfully follow the original plan set by the MoH. According to some respondents, unacceptable criteria such as external monies and influence affected priority setting in 2005-2010 and 2010-2015. The prioritization of HPV was based more on the influence of donors and industry and political sentiment than technical priorities; this contributed to stakeholders' dissatisfaction with the priority setting process. Banura and colleagues 16 13 A lack of participation of key stakeholders contributed to dissatisfaction with the priority setting process for both HPV and PCV. There was a sense that because of a lack of leadership in the MoH, compliance with the priority setting process had decreased and the process had become less consultative, in which the participation of key actors such as MoH staff and the district officers had declined. This was in part explained by the changes in top leadership in the MoH and a reduction in development assistance partners' adherence to a consultative priority setting process. However, despite the dissatisfaction with the priority setting process expressed by many of our respondents in interviews, no dissentions were formally recorded in reports. In some of the newspaper reports, the public reacted negatively to the new vaccine (HPV) but no explicit complaints or revisions to the decisions were documented or mentioned by our respondents. This could be explained by the lack of explicit mechanisms for appealing the decisions.
In terms of the achievement of health system objectives, indicators such as the percent reduction in disability-adjusted life years (DALYs) could not be measured; however at the time of the data collection, the implementation of both vaccines was contributing to the health system's goals of reducing morbidity. Arguably, the implementation of PCV, which was scaled up completely across the country, contributed more to this goal than HPV, which remained at the pilot phase at the time of the study. In terms of increasing investment into and strengthening of the health sector, the priority setting process may have met this parameter; this is because the introduction of the new vaccines involved substantial investment into the health sector and supported other health sector activities such as disease surveillance and capacity building. However, this was specific to EPI. While GAVI expects and facilitates improved financial accountability at the national level, 15 this did not seem to trickle down to the districts where delays in district accountabilities for PCV affected vaccine implementation; although this problem was later rectified. Consistent with the literature, contextual factors impacted the parameters of successful priority setting. 34 Some of the factors, such as strong political commitment to vaccines facilitated the introduction of the new vaccines. However, most of the contextual factors negatively impacted priority setting. For example, the political decisions to change the top management, the influence of donor and industry priorities, and the economic context, which led to a lack of expansion of the budget for the health sector and subsequently resulted in inadequate staff for implementation at the district level, all had a negative influence on priority setting. A few parameters could not be assessed. The priority setting process should increase public awareness of priority setting, involve the increased input of the public and reflect public values, and facilitate public confidence and acceptance of decisions. The assessment of these criteria was mixed. It was not possible to assess increased public awareness of priority setting and increased public confidence in and acceptance of decisions since these parameters necessitated interviewing members of the public. It was also not possible to assess whether the priority setting process reflected public values, since the documents reviewed do not show the exact role played by the public in the priority setting process. Finally it was also not possible to evaluate the efficiency of the priority setting process and decreased resource wastage, parameters that require real-time assessment.

Conclusion
Based on the evaluation framework, there are several areas where concerted efforts are necessary if priority setting for new vaccines is to be deemed successful. First, the prioritization of new vaccines should be based on explicit frameworks. The lack of clarity as to why and how HPV was prioritized over other vaccines that had been identified through a national level process calls for commitment to transparency with regards to how new vaccines get onto the political agenda. HSSIPs should be used as the basis for implementing the documented new vaccine priorities. There is also a need for enforcement mechanisms to ensure that vaccines identified as a priority in the initial HSSIP planning process are not overtaken by vaccines that are not considered to be a top priority. This requires the direct involvement of key stakeholders such as district officers in determining national level priorities. The process should also require public consultations and involvement. This would call for raising public awareness about the priority setting process and the rationale behind the decisions. Monitoring implementation of the new vaccines should also involve monitoring the public response to the set priorities and their implementation. The existing power imbalances between sub-national governments, national governments and their development assistance partners need to be mitigated by strengthening institutional capacity and leadership; this is essential to ensure that priorities are identified and implemented through an inclusive, transparent, and efficient process.
To a great degree, Kapiriri and Martin's framework and sources of information 6 provided viable guidance for evaluating the degree to which the priority setting processes for identifying new vaccines worked in Uganda. The results re-emphasize the need for evaluation to be built in the priority setting process a priori, and that information on priority setting is gathered throughout the implementation cycle as opposed to the traditional ex-ante evaluations. 5