Evidence for Informing Health Policy Development in Low-Income Countries (LICs): Perspectives of Policy Actors in Uganda

Background: Although there is a general agreement on the benefits of evidence informed health policy development given resource constraints especially in Low-Income Countries (LICs), the definition of what evidence is, and what evidence is suitable to guide decision-making is still unclear. Our study is contributing to filling this knowledge gap. We aimed to explore health policy actors’ views regarding what evidence they deemed appropriate to guide health policy development. Methods: Using exploratory qualitative methods, we conducted interviews with 51 key informants using an indepth interview guide. We interviewed a diverse group of stakeholders in health policy development and knowledge translation in the Uganda health sector. Data were analyzed using inductive content analysis techniques. Results: Different stakeholders lay emphasis on different kinds of evidence. While donors preferred international evidence and Ministry of Health (MoH) officials looked to local evidence, district health managers preferred local evidence, evidence from routine monitoring and evaluation, and reports from service providers. Service providers on the other hand preferred local evidence and routine monitoring and evaluation reports whilst researchers preferred systematic reviews and clinical trials. Stakeholders preferred evidence covering several aspects impacting on decision-making highlighting the fact that although policy actors look for factual information, they also require evidence on context and implementation feasibility of a policy decision. Conclusion: What LICslike Uganda categorize as evidence suitable for informing policy encompasses several types with no consensus on what is deemed as most appropriate. Evidence must be of high quality, applicable, acceptable to the users, and informing different aspects of decision-making.


Implications for policy makers
Policy development requires different types of evidence and there is no single type of evidence agreed upon as the most appropriate. • Different stakeholders attach more importance to different types of evidence. • Policies whose implementations impact on several institutions and systems are likely to require more evidence, compared to those where implementation only requires minimal adjustments in the current practice. • The quality of evidence is a very important aspect which creates more confidence in the results, and increases the likelihood of knowledge translation. • In order to ensure that policy-makers receive evidence that meets their expectations, information on their needs is necessary. This could be in the form of national research agendas and/or, needs assessment findings. Such an initiative would need to be routine and the needs should be updated regularly involving a dialogue between research producers, and policy-makers as research users.
action that policy or decision-makers have already chosen to take (symbolic use); may be used to inform debates and modify the way decision-makers think and see a given issue (conceptual use); or may be directly applied in practice or policy (instrumental use) (1).
The definition of what this evidence is, is still a topic of debate with policy-makers, researchers and other stakeholders agreeing very little on what constitutes it (2). In this article, we adopt the definition of evidence as provided by Lomas et al. (3) defining evidence as facts (actual or asserted) intended to help the reader reach a conclusion and form an opinion about something. We acknowledge that evidence and the process of translating it into a decision or some form of action is shaped and is influenced by a number of factors which have been explored extensively by several scholars (4)(5)(6)(7). For example, the Overseas Development Institute's Research and Policy in Development (RAPID) framework in particular looks at these influences with a LIC perspective. It acknowledges that literature on the research-policy link is increasingly moving away from older assumptions that looked at research to policy as a linear process involving two distinct communities of researchers and policy-makers dealing with a specific set of findings deemed as 'knowledge' (7). Instead this linkage is now viewed as a dynamic and complex process that is shaped by stakeholder interactions, the political context, the evidence available, and several external factors. Given these influences, what is deemed as evidence may indeed differ in time, place and in a given situation.
Although there is a general agreement on the need for, and benefits of using evidence to inform policy development and practice, and there are multiple forms of evidence to provide guidance, what, and how much of this guidance counts as evidence, is still not very clear (8,9). The discussion regarding suitable evidence to guide policy development is a long standing one with researchers generally arguing in favor of clinical trials and systematic reviews (9,10). Pang however differs from this argument stating that in the case of developing countries, evidence needs to be broader than that based solely on randomized controlled trials (11). Furthermore, Ritter's findings on how policy-makers access evidence when faced with a decision-making opportunity (12) show that in fact Pang's argument may not apply to developing countries alone highlighting the fact that, while policy-makers are reading and reviewing research, they are at the same time looking at other aspects to decision-making like political viability and the degree of community support. Categorizing the quality of evidence is another aspect where researchers and policy-makers hold different views. While the former attach a lot of importance to methodological rigor (2,13), the latter look for important information based on quick reflections of the realities of policy and decision-making in political and social contexts (2). A systematic review by Invaeer et al. (14) found that one of the facilitating factors for Knowledge Translation (KT) was whether the available was deemed to be of good quality. Noteworthy is the fact that a number of frameworks to standardize the definition of what good quality is -which is very subjective -have been developed; for example, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach (15). The challenge is the ability to apply these frameworks given the fact that policy-makers, who are usually not equipped with skills to use these frameworks, will determine the quality of the evidence that they have available. Other characteristics of the evidence that have been shown to favor KT include the timeliness and relevance of the evidence, and the credibility of researchers (16). Furthermore, the question regarding what evidence is enough continues to be debated. There have been suggestions that the kinds of evidence relevant for the different stages of the policy process do in fact differ (17,18). Therefore, what accounts as evidence does in fact vary through the policy cycle. Participation of the sub-national level, CSOs, PNFPs and PFP is through representation. Representatives are selected through a consultative process within the constituents and they collect views from constituency members prior to participation in meeting, as well as providing feedback regarding decisions taken.

Methods
Getting evidence into policy is a complex process which occurs amidst multiple stakeholder interactions embedded in a given context and as such, we employed mixed methods to capture complexity, enhance comprehensives and validity of our results (19). Exploratory qualitative methods which are well-suited to capture complexity (19) were employed to explore views of respondents, on what evidence they deemed appropriate to inform policy development. Quantitative methods were employed to assess the frequency with which different types and characteristics of evidence were mentioned by the respondents. We conducted interviews with key informants using an in-depth interview guide. The study was conducted between June, 2012 and August, 2013.

Selection of respondents
Respondents were selected from the national and subnational (district) levels. We developed a sampling frame based on work done earlier that identified stakeholders in health policy development and KT in Uganda (20). At the national level, identified stakeholders included the MoH, Ministry of Finance (MoF), donor representatives, media, CSOs, PNFP and PFP service providers, researchers and parliamentarians. Within the identified institutions, respondents were purposively selected based on the level of seniority in their institutions whereby the most senior officers were selected as these were more involved in health policy development and evidence generation and; having knowledge about the research question (21). Researchers were selected on the basis of their previous work on KT and focus on health system development, while the journalist was selected based on focused reporting on health issues. In the case of CSOs, PNFP and PFP, participation in the policy process is by representations and the representatives of the umbrella organisations were selected. The parliamentarian was a member of the social service committee which deals with health issues. At the sub-national level, two districts were selected based on proximity to the capital city for easy access given resource constraints and, presence of a regional referral hospital (Jinja district) or general hospital (Mpigi) to obtain perceptions from across the spectrum of healthcare delivery system. At the sub-national level, management of health services within the district is the responsibility of the District Health Management Team (DHMT) headed by a District Health Officer (DHO). District health managers and service providers participate in policy development through representation. They also participate in generation of evidence that informs health policy and strategy development in addition to using evidence at the local level to improve service delivery. At the sub-national level, the DHO and a member of the DHMT in charge of supervising health facilities within the district were purposively selected and interviewed. Within the selected districts, two hospitals and two lower level facilities (one public and one PNFP in both cases) were purposively selected based on proximity to the district headquarters for easy access, and our desire to capture the different levels of the healthcare system. The medical superintendent or health center employee in-charge and one clinical staff member responsible for the outpatients department were purposively selected and interviewed at each health facility. Detailed information regarding the selected KI is shown in Table 1.

Data collection
An interview guide, comprising of open-ended questions was developed to explore the views of respondents regarding suitable evidence to guide policy development. The interview guide was developed by the research team and pre-tested with volunteer colleagues in the MoH (n = 2) and the World Health Organization (WHO) Uganda office (n= 2). The research team refined the guide prior to undertaking interviews. Respondents were invited by telephone to participate in the study and all interviews were conducted face to face by the first author. All invited respondents agreed to participate in the study.

Data analysis
Interviews were recorded verbatim, transcribed the same day and entered into Microsoft Word. Additional notes taken by the research team during the interviews were used to enrich the transcribed interviews. Interviews lasted on average 30 minutes. As a first step to formal analysis, the first author read through all the interviews and identified emerging issues in line with the study objectives. The research team then read all interviews to identify emerging issues by type of KI. Inductive manifest and latent content analysis techniques (15) were undertaken in QSR NVivo 10. Manifest content analysis (22) was initially undertaken to assess the visible meaning of the text and following which, the underlying meaning was explored through latent content analysis (22) an example of which is shown in Table 2. Converging issues were reviewed by the research team and where interpretation differed, consensus was achieved through revisiting the raw data. Converging issues were grouped under the different themes. The quotations that further highlight emerging issues were edited to ensure clarity and used where appropriate.

Results
Responses regarding what type of evidence was deemed suitable for informing policy development were categorized under several themes that emerged based on content analysis namely: 1) high quality evidence covering different aspects; 2) local research embedded in local context; 3) routine Monitoring and Evaluation (M & E); 4) international evidence; 5) clinical trials and systematic reviews; 6) observational reports from service providers; 7) community reports/cries; and 8) experience, as shown in Table 2. From the quantitative results, the frequency with which different types and characteristics of evidence were mentioned by the different respondents is shown in Table 3. High quality evidence, informing different aspects of decision-making was deemed as important given the fact that development of successful policies covers several parameters. Among the aspects highlighted by the respondents were, evidence on cost to assess whether the policy will be affordable to the government, community acceptability to assess uptake of the policy, cost effectiveness to assess value for money, health system capacity to assess implementation feasibility and political acceptability.  Code Categories Theme -We need a combination, no one source could be the most used -We need high quality research on different aspects, clinical, social sciences, costing, implementation feasibility, community acceptability -Evidence on several aspects clinical, costing and supply implications, implementation feasibility -You need a combination of data but high quality data. If its studies they should be of good methodology, appropriate sample size and if it is routine data it should be data that is consistent -I think every piece of research needs to be considered. The key thing that the policy-maker needs to do is to decide which evidence is most appropriate for the local context Different aspects of evidence play different roles and all are of relevancy to policy development High quality evidence covering different aspects of decisionmaking Preference for local research because it depicts the background of the exact factors -social, biological -of that locality -Local studies may have more relevance to Ugandan's policy-making -Local-real life research will impact more on decision-makers -You need local evidence about which medicines are effective for the local context -Need for local context data, evidence on costs, implementation feasibility, compliance -International studies need to be considered as well but alongside local data to make sure it fits into the context -Need both real life local research evidence in addition to the international evidence about drug efficacy, compliance International evidence used alongside local data Contextualized international evidence -Clinical trials, drug efficacy studies especially these ones which compare existing drugs with those that have been proposed for adoption -Systematic review of different papers if various papers published from different settings are all speaking about the same thing, this is a good source of information -Systematic reviews which put together different small studies to get a bigger picture that can give you even better more precise information Decision should be based on high quality studies carried out in different settings, in well-controlled settings e.g. clinical trials, drug efficacy studies, systematic reviews Clinical trials and systematic reviews research would depict the background of the exact factorssocial, biological, or political of that locality. International research may complement but most important to me is the local research done within the country which gives the exact picture of that country". Although international evidence plays a role, it must be contextualized and triangulated with what locally generated evidence has shown. M & E was noted to play a key role in some instances for example, monitoring implementation of policies and among the noted advantages was the ability to provide data on the situation on the ground. A parliamentarian remarked that: "People are persuaded by M & E because the question they want answered is how are polices being implemented and whether we are achieving results". Some respondents however cautioned against the weaknesses in the routine M & E systems and emphasized the need to undertake research to verify reported achievements and observed disease trends. A CSO respondent remarked that: "There are times we take decisions based on M & E which is good because it gives you trends on how things are happening but sometimes to nail that harder, you may need to look at some studies that go further to confirm observed trends, but also to highlight the weaknesses of our M & E Systems". Researchers showed a bias towards clinical trials and systematic reviews stating that this is the most credible source. A researcher remarked that: "Being a scientist the best type of evidence is a well-conducted randomized clinical trial, this should be the best evidence on which to base our policies. We need several clinical trials and if you get consistent results from more than one trial then that is the best evidence scientifically, and from my stand point". Community voices are also evidence that can be used in policy development as remarked by a journalist in reference to policies on health financing: "There was evidence from the ground that very many people were not able to afford user fees and as a result, they were staying away from the health facilities even when they were sick. That was definitely evidence in the form of peoples' testimonies".
"Experience" was also identified as a form of evidence that can guide policy especially in instances where there is nothing to compare with. A researcher stated that: "You may find that in some cases, the best evidence is experience. You need somebody who has experience in that particular area to guide you; you don't need to commission a study". Different types of polices will require different evidence as remarked by a parliamentarian that: "Each policy issue requires a certain type of evidence". The extent of risks associated with a given policy has an impact on the nature and scope of evidence required. Policies which have wider health system implications, call for significantly more resources, affect more people and where the risks of success or failure are not very clear, are likely to require more evidence on the different aspects. A researcher stated that: "If you are going to start a social health insurance scheme, the fund requires major policy action and you are going to change so many things so you need evidence on several aspects. Whilst policies like giving septrin to people who are having HIV would not require much evidence because septrin is available in the system and people are already diagnosed with HIV, all they need is medicine".

Discussion
This study has shown that there is neither a single type of evidence agreed upon as the most appropriate to inform policy development, nor is there any that is deemed best for stakeholders in general. Policy development requires several types of evidence and the different stakeholders attach more importance to different types of evidence. Furthermore, the quality of evidence is perceived as a very important attribute providing more confidence in the results and enhancing the likelihood of KT. Our study has highlighted the need for evidence informing different aspects of decision-making including political and community acceptability, affordability and implementation feasibility among others. This mirrors the fact that policymakers look for evidence that covers different aspects that they are concerned with in the decision-making  process including factual information, but also context and feasibility aspects of a decision. It is not enough that an intervention is effective, policy-makers want to know how applicable it is and whether it is affordable and acceptable by the community. Policies with far reaching consequences require more evidence compared to those with minimal requirements for implementation and adjustments of current practice. Mubyazi and Gonzalez-Block (8) (24). Sutcliffe and Court (24) argue that at the agenda setting stage, emphasis is on raising awareness and evidence highlighting the magnitude of the problem may play a key role in decisionmaking. At the policy formulation stage on, there is need to understand the situation and the different options and as such, comprehensive evidence linking activity and an outcome, as well as the expected cost and impact of an intervention will be required. The implementation stage on the other hand will require operational evidence, systematic learning and evidence from pilots which demonstrate how to implement the policy. We however highlight the iterative nature of policy development and as such, demarcations in the policy cycle are not distinct. For example, Nanyunja et al. (25) documented a case of the malaria treatment policy change in Uganda where even after a decision had been made on which first line antimalarial to adopt, engagement of some actors with decision-makers continued and indeed an alternative first line treatment was also adopted. Our study has highlighted the importance of evidence being of high quality which is already a documented facilitating factor for KT (16,26). The definition of high quality may be a matter to be further explored but it is evident that policymakers are aware that not all evidence presented to them is credible, robust and of value. Evidence needs to win the trust of the decision-makers if it is going to be considered for use in the policy-making process. Different stakeholders lay emphasis on different kinds of evidence for varied reasons. For example, while most donors preferred comprehensive and international evidence and MoH officials looked to comprehensive and local evidence, most district health managers preferred local evidence, routine M & E reports and observations from service providers. Service providers on the other hand preferred local evidence and routine M & E whilst researchers preferred systematic reviews and clinical trials. Two reasons may explain this observation; firstly, expressed preferences may be related to the stages of the policy-making process that actors are more involved with. Secondly, preferences may be related to the level of decision-making they are a part of, that is, subnational, national or international. In the past, researchers have been criticized for their preference for, and focus on scientific evidence because of its rigor, and this has formed a point of diversion between them and policy-makers who are looking for quick solutions to address the time bound nature of policy development (2). Researchers are however fast recognizing that research evidence is just one source of information for policymakers (27). Anecdotal evidence however shows that many researchers still hold the notion of evidence being in a constant hierarchy with non-peer reviewed evidence being ranked lower than its peer-reviewed counterpart. However our study shows that such ranking may indeed not be a real reflection of what different policy-makers deem important at any one given decision-making time. In a systematic review by Oliver and colleagues (6), the researchers pointed out that over a third of the studies they included cited policy-makers using evidence other than that from peer reviewed research, for example, local data and tacit knowledge. In this study, we formed categories of the types or forms of evidence respondents deemed important. These categories include research in addition to other types evidence among which is routine M & E, service provider reports, community feedback and experience. These categories fit closely with those cited by a systematic review by Orton et al. (28) that aimed to synthesize empirical evidence on the use of research evidence by public health decision-makers. They found that in addition to primary studies and systematic reviews, the types of evidence used included, internal program evaluations, local and provincial best practices, and natural policy experiments among others. In addition the categories tally with the five types of evidence influencing policy on health inequalities that Whitehead and colleagues (29) identified as particularly persuasive with policy-makers. These included observational evidence showing the existence of a problem; narrative accounts of the impacts of policies from the household perspective; controlled evaluations; natural policy experiments; and historical evidence. The emphasis on research evidence, systematic reviews and randomized controlled trials as the ideal evidence in several studies in the past may be a reflection of the fact that very few studies had explored the views of policy-makers on the subject of what evidence is, and their day-to-day use of it in policy and decision-making (28). Policy-makers seek robust dialogue, critical and creative thinking, and will value tacit knowledge like expert opinion alongside research which may often be necessary but insufficient for decision-making (27). This is a view that is shared by several others including Pang (11) who points out that policy-makers for whom access to the most relevant and useful evidence is crucial, are constantly seeking to answer the questions of whether an intervention can work and whether it is worth it. These questions can only be answered by evidence from a combination of sources. In our study, local evidence ranks highly for local policy and decision-makers and, evidence from experience is valued too. This may highlight the fact that evidence should be applicable and acceptable. Several studies involving policy-makers' views have emphasized this (28) and in fact, Quazi et al. (30) advise that researchers must understand that the evidence they generate needs to be contextualized for applicability and utility. Indeed, it is advised that evidence be interpreted with the backdrop of the local context in which the research was carried out (31). There is an ongoing debate about whether differences exist between LIC and High-income Countries (HICs) policy processes and their evidence needs. Lavis and colleagues have argued against this (32). They believe that the processes are generally similar only differing in terms of context, which may be due to systems' issues. On the other hand, some scholars believe that these differences are indeed real (33). This would explain why LICs' views regarding what evidence is, differs from developed countries' views. This is perhaps enhanced by the uncertainties involved given the weak health systems, resource constraints, and population demands faced by politicians in LICs which may influence the nature of the evidence required.

Implications for research
This study highlights the importance of policy-informed research agendas. In as much as policy should be informed by research, researchers need to be informed by policy in order to carry out relevant and context sensitive research, if they are going to be of value to the policy-making process. Literature highlights that the involvement of policy-makers in research is still sub-optimal. For example, Oliver et al's systematic review of 145 studies found 137 study reports were written by researchers or people with academic affiliations, while policy-makers were credited as authors in only 3 studies, one of which was a governmental report (6). This is a reflection of the suboptimal involvement of users in the production process of research. This trend may need to change; the involvement of policy-makers makes the research a tad nearer their needs and expectations. Furthermore, there is a global call for research agendas to be informed by policy. It is waste of resources when research is done but not able to inform policy or practice, or form a base on which other policy relevant research can build. This is especially so in LICs where resources are extremely meager and the luxury of doing research for the sake of it may not be affordable. Hutchison and Carden advise that if researchers want to design and implement influential research, they need to understand the settings in which the policies they intend to inform are made such that, their research strategy aims for the best possible effect within that setting (31,34). In addition, the call for a combination of different types of evidence, which implies that research evidence is, but only one of several types, calls for researchers to align themselves with the policy development process. A lot of non-peer-reviewed research is commissioned by government departments to inform decisions and policies while a lot of the peer-reviewed research is frequently done uninformed by the policy process, thereby being released at a time it is not of interest no matter how fascinating the results are. In such instances, the non-peer-reviewed research would have higher chances of informing the policy and decision-making processes as compared to peer-reviewed scholarly research. Researchers need to endeavor to inform themselves of the stages of policy development, and combine this with knowledge on the priorities of the policy agenda as they design their research. Despite being close to the problems and situations, and the context in general, research capacity in LICs is low in all aspects (35). Some of this weakness is owed to a limited number of skilled persons to carry out research, a lack of research infrastructure, overdependence on donor funding for research activities, uncoordinated initiatives, inadequate political will, the transfer of data from Low-and Middle-Income Countries (LMICs) to developed countries, publication research without any feedback to policy-makers in the source countries, and a focus on research that offers huge financial returns on investment, instead of topics that are more locally relevant (36). Considering the preference for local evidence, this means there is a call for more research to be carried out locally. The inter academy medical panel notes that national and regional science and medical academies are better placed than foreign institutions to influence policymakers with evidence, and to support research capacity building in countries and regions where they operate (37). Given the weak research capacity in LICs, innovative ways to meet this need are demanded. North to South cooperation is inevitable but so is South to South collaboration, to avoid duplication of activities and have more coordinated initiatives, and furthermore share resources like equipment at given centers of excellence.

Implications for policy
The findings of this study highlight some implications for policy-makers and stakeholders. In order to ensure that policy-makers receive evidence that meets their expectations, information on their needs is necessary. This could be in the form of national research agendas and or, needs assessment findings. Such an initiative would need to be routine and the needs updated regularly involving a dialogue between research producers and policy-makers as research users. Policy-makers have expressed a leaning towards non-peerreviewed evidence, the lack of skills to make optimal use of research evidence may explain this observation. This would represent a missed opportunity for research evidence to inform the policy-making process. It would at the same time highlight the role of knowledge brokers or KT platforms which are national or regional level institutions committed to fostering linkages and exchange across health systems, skillfully acting as intermediaries between research and policy (38)(39)(40).

Strengths and weaknesses of the study
Our study has several strengths; firstly, we interviewed a wide and diverse group of stakeholders allowing for a broad picture on the subject. Secondly, the diverse nature of the respondents provides a rich multi disciplinary and multi institutional perspective on evidence deemed appropriate to inform policy decisions. Our study has limitations as well among which is the social desirability bias where respondents report what they think the researcher would like to hear and give responses displaying 'acceptable' attitudes and behaviors. However this may not have been a big problem as we did not go out to explore how much or what they were doing. Some scholars have noted that different types of evidence are needed at the different stages of policy development but our study did not explore this aspect. We however feel that this limitation does not affect the relevance of our results given the fact that policy development is iterative and in majority of cases one cannot draw clear demarcations between the different stages.

Conclusion
What LICs like Uganda categorize as evidence suitable for informing health policy development encompasses several types and no one source is deemed superior to others. Evidence needs to be holistic, of high quality, applicable and acceptable to the users. Preference for locally generated evidence has been expressed but the quality of which needs be enhanced. In as much as policy needs to be informed by research, researchers also need to be informed by policy in order to carry out relevant and context sensitive research.