An Implementation Research Approach to Evaluating Health Insurance Programs: Insights from India

One of the distinguishing features of implementation research is the importance given to involve implementers in all aspects of research, and as users of research. We report on a recent implementation research effort in India, in which researchers worked together with program implementers from one of the longest serving government funded insurance schemes in India, the Rajiv Aarogyasri Scheme (RAS) in the state of undivided Andhra Pradesh, that covers around 70 million people. This paper aims to both inform on the process of the collaborative research, as well as, how the nature of questions that emerged out of the collaborative exercise differed in scope from those typically asked of insurance program evaluations. Starting in 2012, and over the course of a year, staff from the Aarogyasri Health Care Trust (AHCT), and researchers held a series of meetings to identify research questions that could serve as a guide for an evaluation of the RAS. The research questions were derived from the application of a Logical Framework Approach (“log frame”) to the RAS. The types of questions that emerged from this collaborative effort were compared with those seen in the published literature on evaluations of insurance programs in lowand middle-income countries (LMICs). In the published literature, 60% of the questions pertained to output/outcome of the program and the remaining 40%, relate to processes and inputs. In contrast, questions generated from the RAS participatory research process between implementers and researchers had a remarkably different distribution – 81% of questions looked at program input/ processes, and 19% on outputs and outcomes. An implementation research approach can lead to a substantively different emphasis of research questions. While there are several challenges in collaborative research between implementers and researchers, an implementation research approach can lead to incorporating tacit knowledge of program implementers into the research process, research questions that are more relevant to the research needs of policy-makers, and greater knowledge translation of the research findings.


Background
The emerging field of implementation research seeks to focus research attention on issues concerning program implementation to improve program effectiveness.It emphasizes generating knowledge for action and not just for the sake of 'better understanding.' [1][2][3] One of the distinguishing features of implementation research is the importance given to both the context within which a program operates, as well as, the population that is affected by the program.In particular, it seeks to involve implementers and populations affected by an intervention in all aspects of research right from research design, the process of research, and as users of research. 1,3mplementation research methods, in the spirit of participatory research, emphasize involving 'local' populations and groups in research to enable a 'bottom-up' approach so that local priorities and participants have a voice.This voice is critical to make research, and the action that results from it, more relevant and acceptable locally. 3This requires 'local' participation in the research process, in framing the questions, and as consumers of this research. 3Implementation research places emphasis on including a range of implementation agents -managers, executive decision-makers, policymakers, and practitioners -in the process of identification, design, and process of research. 1Incorporating a program implementer perspective makes the research process sensitive to the complexity of the world that program implementers inhabit and are trying to change.This paper reports on an implementation research approach to evaluating a health insurance scheme in India.Here, program implementers and researchers worked together to define the scope of research and in identifying relevant research questions.It aims to both inform on the process of the collaborative research, as well as, how the nature of questions that emerged out of the collaborative exercise differed in scope from those typically asked of insurance program evaluations.As such, this collaborative effort included the perspective of program managers and not beneficiaries or other stakeholders.The context was one of the longest serving government funded insurance schemes in India, the Rajiv Aarogyasri Scheme (RAS) in the state of Andhra Pradesh. 4 The RAS has been operating since 2007 and covers the cost of inpatient care (up to a limit) for people below the poverty line (for more information on the RAS see http://www.aarogyasri.telangana.gov.in/).The program itself covers around 70 million people. 4,5In this article, we discuss how when an implementation research approach is used, the kind of research questions that emerge offer a more comprehensive view of program performance, and one that is more closely aligned to implementer needs.

The Implementation Research Approach
Starting in 2012, and over the course of a year, staff from the Aarogyasri Health Care Trust (AHCT), the Public Health Foundation of India (PHFI), and the Indian Institute of Public Health, Hyderabad (IIPH), held a series of meetings to identify research questions that could serve as a guide for an evaluation of the RAS.The meetings were held at the AHCT offices in Hyderabad.There were usually between four to six members present, with at least one from each of the three partner agencies.AHCT was represented in the collaborative meetings by their Chief Executive Officer (CEO), who is an officer of the Indian Administrative Service, and members of their research unit, who are medical doctors and statisticians.As such, the top management of the AHCT was involved in the collaborative effort.PHFI and its affiliate IIPH-Hyderabad were represented by researchers with doctoral or masters level training in public health.The process was initiated by AHCT who were planning to commission an evaluation of the RAS.They also proposed that the research questions of interest should emerge from the application of a Logical Framework Approach ("log frame") to the RAS.The job of drafting the log frame was the joint responsibility of PHFI and the AHCT research team.The meetings served as a forum of discussing the log frame in the wider context of AHCT.A log frame is a way of describing the design of a project; it represents the different stages in a project over which activities convert inputs to outputs and outcomes to achieve impact on higher levels of objectives. 6At the heart of the log frame is a matrix.While there is some variation in how the matrix is structured, a common way is to have the rows of the matrix represent a logical hierarchy of project activities, outputs, purpose and overall objectives, such that, the fulfillment of a lower level element leads to the achievement of a higher level one. 6The columns of the matrix typically contain indicators to measure performance of each row element and sources of information on, and assumptions required for, achieving the indicator targets.The purpose of developing the RAS log frame was to delineate the scope and identify relevant questions for a comprehensive evaluation of the RAS.As such, it adapted the log frame matrix so that the overall goal of the RAS was decomposed into subobjectives and each related to program outputs and activities (input and processes) (Table 1).The starting point of the log frame was the stated objective of the RAS -"To provide free quality hospital care to poor families to protect against catastrophic health expenditures through strengthened public institutions and purchased private services." Three thematic components (ie, sub-objectives) encompassed this objective: (i) Coverage, access and equity; (ii) Health financing; and (iii) Effectiveness of services procured.Outputs and related input and process activities that contributed to achieving these outputs were identified.Table 1 describes the goals, components and related activities of the log frame.The matrix presented in Table 1 was the basis of identifying research questions for evaluating the RAS program.The outputs and activities easily lend themselves to being framed as research questions.For example, the program output of ensuring that all poor families in the state had a beneficiary card was translated into the research question "Do all households below the poverty line have a beneficiary card?"Similarly, the program process -conduct Information Education Communication (IEC) activities to improve awareness of RAS (1.2.1 in Table 1) translates to the question "Have RAS beneficiaries heard about the scheme and know its features?"AHCT intended to uses these research questions to guide the scope of RAS evaluations.The process of linking the RAS insurance scheme's outputs to activities also focused attention on the interrelatedness of different aspects of the programme.For instance, in the component evaluating the effect of RAS on reducing outof-pocket (OOP) health expenditures, a set of questions relating to activities which linked relevant program inputs (such as offering free diagnostic and outpatient services) and processes (outreach camps) to this objective were identified.Consequently, a diverse and comprehensive set of questions on the role of health workers, capacity building, community awareness, processes of enrolling beneficiaries and other aspects of input and process elements needed to be considered.

Comparisons With Evaluations of Health Insurance Programs
Implementation research places considerable emphasis on involving program implementers, managers, and policymakers as partners in identifying, designing, and as users of research.Over the last two decades, several low-and middle-income countries (LMICs) have invested substantial resources in expanding national health insurance programs. 7,8esearch on the performance of these insurance programs have largely focused on their impact on outputs and outcomes such as financial protection, utilization of health services, social inclusion, resource mobilization, quality of care, and community empowerment. 9The previous section described the process through which researchers and RAS administrators collaborated to set the scope and identify research questions for evaluating RAS using the log frame.Of interest is to see how the nature of questions that emerged out of this implementation research approach differed in focus compared to those found in the literature on evaluation of health insurance programs in LMICs.We conducted 11 searches for papers on the evaluation of health insurance schemes across LMICs, indexed in the National Institutes of Health's database, PubMed.It is important to note that our purpose here was not to conduct a comprehensive review of evaluations of health insurance schemes.Rather, it is to review recently published evaluations of health insurance programs in LMICs and classify the focus of their research questions.The search was restricted to papers published between the years 2010 and September 15, 2015.Search strings included the following keywords: health insurance, LMIC, evaluation, low-and middle-income countries, insurance, impact, health insurance scheme, and social health insurance schemes.A total of 127 papers were obtained through the 11 searches conducted on PubMed.Of these, 55 duplicate results were removed, the rest were scanned for relevance to the subject on the basis of which an additional 47 articles were eliminated.The inclusion criteria were studies on LMICs.Studies included those on the evaluation of health insurance, implementation, policies on health insurance, and descriptive studies on health insurance.The exclusion criteria were non-insurance linked studies on health expenditure (example, studies on household expenditure on health or expenditure on management of chronic conditions), studies on health insurance in highincome countries and those published before 2010 or after 2015.The remaining 25 relevant papers which emerged from this exercise were reviewed on the basis of the objective of the study and research question.Among the 25 relevant papers, two were systematic reviews (Spaan et al 9 and Faden et al 10 ).Studies included as part of these two reviews were snowballed for our review.After excluding duplicate entries, 140 papers (80 from Spaan et al 9 and 60 from Faden et al 10 ) were retained.The majority of these, 128 papers (peer-reviewed and grey literature), were published before between 1990 and 2009.Thirteen peer-reviewed papers met our inclusion criterion, of which one paper was reviewed in the preliminary search; therefore, 12 papers were included in the review, giving us a total of 37 papers.Their research questions were classified according to their focus on input, process, or outcome/ outputs of the insurance scheme they were evaluating.The papers reviewed on evaluations of health insurance schemes in LMICs covered countries across Europe (Georgia and Moldova), Asia (India, China, and Thailand), Africa (Nigeria, Kenya, Ghana, Tanzania, Namibia, Rwanda, Uganda, and Burkina Faso), and Mexico from the Americas.The majority of papers reviewed, focused on national and social health insurance schemes with the policy and systematic reviews used in 9 of the 37 papers reviewed.We searched the papers for evidence of participatory research -only two papers used participatory methods; this was to examine issues of access and implementation towards improving access and quality of health services.The majority of questions addressed by the studies focused on program outputs/outcomes, such as, the effect of OOP expenditure on healthcare and utilization of services (Table 2).A single publication could have questions that focus on more than one of these three categories.Table 2 shows the distribution of question identified from the literature review and those from the collaborative exercise classified according to their focus on input/process, or output/ impact aspects of the insurance scheme.Questions identified from the literature review covered input/process, and output/ outcome aspects of insurance schemes.Input/processes refers to resources (human, physical, financial), procedures, and activities that make insurance programs functional.Output/ outcomes refer to what is produced by these input/process Conduct death audits Abbreviations: RAS, Rajiv Aarogyasri Scheme; PHC, primary health centres; CHC, community health centres; NWH, network hospitals; OOP, out-of-pocket; IEC, information education communication; NAMs, Network Aarogyamitra; OPD, outpatient department; IPD, inpatient department.a Goal: Provide free quality hospital care to poor families to protect against catastrophic health expenditures through strengthened public institutions and purchased private services activities.The majority of questions on insurance schemes pertained to output/outcome measures (Table 2).Around 60% of the questions in the published literature pertained to program outputs and outcomes, and the remaining 40% asked about program input/process.In contrast, questions generated from the collaboration between RAS administrators and researchers in the implementation research approach had a very different distribution -almost 81% of questions asked about input/processes, and 19% focused on outputs and outcomes.

Discussion
Implementation research emphasizes engaging with program implementers in all aspects of research.It represents an attitude or approach in which power in the research process is shifted from researchers to a partnership between researchers and implementers. 1,3Its parallel in participatory research is the realignment of power in the research process to include communities rather than the conventional research model where power is vested mostly in the researcher. 3Engagement with program implementers offers several potential benefits.It makes research more relevant to the needs of implementers since they are involved with identifying research questions.The research process can also be richer by incorporating the tacit knowledge of program functioning that implementers possess.Perhaps most important, the results of research are more likely to be accepted and acted upon by implementers when they have ownership over the process and product.As the RAS research collaboration shows, involving program implementers and researchers in determining the scope of research can lead to research questions having a substantively different emphasis compared to those typically published in the literature.While all research on program implementation involves some degree of engagement with program implementers, it is likely to be minimal when there is no deliberate attempt to engage implementers.The studies evaluating health insurance programs that we reviewed were researcher driven (except for two) and indicate a strong tendency to evaluate insurance programme against a set of outcomes, largely, on financial risk protection, and to a lesser extent, on process and input aspects of the program.When implementers are made part of the process of identifying research questions, as the RAS collaboration illustrates, the resultant set of questions are much broader in scope and put more emphasis on processes and inputs.Moreover, it also informed implementers and researchers about the causal interlinks between the input, process, and outputs of the program and the program goals.Our experience with the RAS collaboration also highlighted the challenges in engaging in implementation research.Control over the objectives and process of research is always contentious since implementers and researchers may not always give importance to the same issues.Implementation research involves passing judgment on the current state of implementation, and by association, on the implementers themselves.This can lead implementers to prefer focusing on aspects of program evaluation like processes where they have more agency, than on program outcomes that are determined by myriad factors.Implementers may not always appreciate the research process and the sanctity of adhering to research methods.On the other hand, researchers are also not always appreciative of the implementer impatience for getting quick feedback on program performance.The participation and interest of implementers in the research might also change during the course of the research. 3What started out as a participatory implementation research process could then regress into a researcher driven process.While we tend to refer to implementers as a single entity, they are a heterogeneous group.For one, policy-makers, program managers, and health workers are all implementers. 1However, their position allows them to engage with a program in different ways and so their perspectives can be quite diverse.In any program, the management is likely to have different priorities and views than health workers closer to the ground.The natural tendency, as was in the RAS collaboration, is for researchers to engage with program managers since they usually are the 'face' of the program.As such, this represents collaborating with one set of implementers with a specific perspective and experience.Collaborating with another set of implementers (eg, health workers) could result in other types of research questions being emphasized.Despite the many challenges of collaboration between researchers and implementers, an implementation research approach can lead to several desirable outcomes.It can give insight into the functioning of programs that is difficult for external researchers to independently achieve, incorporating tacit knowledge of program implementers into the research process, research questions that are more relevant to the research needs of policy-makers, and greater knowledge translation of research findings.

Table 1 .
Goal, Components, and Activities in the Log Frame a Provide free outpatient services at health camps and NWH 2.1.3.Register RAS eligible cardholders through NAMs upon entry 2.1.4.Provide free and separate outpatient services for RAS beneficiaries at network hospitals 2.1.5.Offer free diagnostic services to RAS beneficiaries at network hospitals 2.1.6.Provide end-to-end (from reporting time to 10 days post discharge) free service for RAS patients 2.1.7.Provide free follow-up services to RAS patients

Table 2 .
Distribution of Research Questions Based on Log Frame Exercise and Published Evaluations (2010-2015) of Insurance Schemes from LMICs

Question Category Examples of RAS Log Frame Question and Their Frequency Published Evaluations From LMICs (2010-2015)
Input/ProcessAre assistants [to facilitate access to network hospitals] present at all PHC and CHC?How long did it take a beneficiary to get a RAS card?What is the time taken for claim settlement?Abbreviations: RAS, Rajiv Aarogyasri Scheme; LMICs, low-and middle-income countries; PHC, primary health centres; CHC, community health centres.