Evidence-Informed Deliberative Processes for Health Benefit Package Design – Part II: A Practical Guide

Background: Countries around the world are using health technology assessment (HTA) for health benefit package design. Evidence-informed deliberative processes (EDPs) are a practical and stepwise approach to enhance legitimate health benefit package design based on deliberation between stakeholders to identify, reflect and learn about the meaning and importance of values, informed by evidence on these values. This paper reports on the development of practical guidance on EDPs, while the conceptual framework of EDPs is described in a companion paper. Methods: The first guide on EDPs (2019) is further developed based on academic knowledge exchange, surveying 27 HTA bodies and 66 experts around the globe, and the implementation of EDPs in several countries. We present the revised steps of EDPs and how selected HTA bodies (in Australia, Brazil, Canada, France, Germany, Scotland, Thailand and the United Kingdom) organize key issues of legitimacy in their processes. This is based on a review of literature via PubMed and HTA bodies’ websites. Results: HTA bodies around the globe vary considerable in how they address legitimacy (stakeholder involvement ideally through participation with deliberation; evidence-informed evaluation; transparency; and appeal) in their processes. While there is increased attention for improving legitimacy in decision-making processes, we found that the selected HTA bodies are still lacking or just starting to develop activities in this area. We provide recommendations on how HTA bodies can improve on this. Conclusion: The design and implementation of EDPs is in its infancy. We call for a systematic analysis of experiences of a variety of countries, from which general principles on EDPs might subsequently be inferred.


Background
Countries around the world are rethinking the design of their health benefit packages to achieve universal health coverage. 1  assessment (HTA) bodies which support governments in these choices. 4 HTA determines the value of a health technology and can inform decisions at different levels, e.g. reimbursement decisions on a single health technology or regarding larger parts of the benefit package.
Increasingly, decision-makers are urged to organise fair, legitimate processes in health benefit package design, with legitimacy referring to the reasonableness of decisions as perceived by stakeholders. 5,6 Evidence-informed deliberative processes (EDPs) were developed in response to this. 7 An EDP is a practical and stepwise approach for HTA bodies to enhance legitimate health benefit package design based on deliberation between stakeholders to identify, reflect and learn about the meaning and importance of values, informed by evidence on these values. In EDPs, the concept of legitimacy is translated into four elements: stakeholder involvement ideally operationalised through stakeholder participation with deliberation; evidence-informed evaluation; transparency; and appeal. The underlying idea is that HTA bodies integrate these elements into the various steps of their processes ( Figure   1). 8 It is hereby important to recognise that the EDP-framework is a holistic approach to enhance legitimacy, even though its terminology may suggest a narrow focus on deliberation.  public as part of their HTA processes. 9 However, this still does not mean that stakeholders are actively engaged in deliberations and can openly exchange views on argumentation and evidence. In other countries, stakeholder involvement is only in its infancies or not organised at all. Similar observations can be made for the other elements of legitimacy (evidenceinformed evaluation; transparency; and appeal) -i.e., HTA bodies vary widely in how they incorporate these into the various steps of their processes. 10 This paper reports on the further development of practical guidance on EDPs to support HTA bodies on how they can best address legitimacy in the various steps of their processes. The first EDP guide was developed in 2019; 11 the updated and comprehensive guide is released at the HTAi meeting in June 2021 and available elsewhere. 9 The guide provides practical recommendations on how a country can improve its HTA process, taking into account that each country has a unique decision-making context and should make its own choices as to what is appropriate As such, it is not meant as a blueprint, but as an inspirational and practical tool. EDPs are currently employed by national health authorities in Ghana, Iran, Moldova, Pakistan and Ukraine for revision of their health benefit packages, 12 and its principles were previously applied for similar use in Kazakhstan, Thailand, 13 the Netherlands 14 and Indonesia. 15 A companion paper reports on the development of the conceptual framework of EDPs. 8 This paper first describes the development process of practical guidance on EDPs.
Subsequently, for each step of the EDP framework, we provide specific guidance on key issues of legitimacy and present a novel overview on how eight relatively well-developed HTA bodies around the world (in Australia, Brazil, Canada, France, Germany, Scotland, Thailand and the United Kingdom) have made choices regarding these issues. We conclude with several overall recommendations. We speak of 'HTA' when referring to the whole process, while 'hta' specifically refers to the evaluation of a single technology.

Development of practical guidance
In his commentary on EDPs, Culyer describes the development of deliberative processes: "the understanding how best to make arrangements (..) immediately takes one to a highly complex academic and professional crossroads of behavioural science, governance, political philosophy, political science, the law, administrative theory, industrial economics and communications. This lattice of disciplines and professions militates against there being any single unifying "theory of deliberative processes" so one needs to add other requirement:  requires imaginative work by people well-grounded in the practical realities of their own culture and politics and a systematic accretion of descriptive material from which, over time, one may be able to infer some general principle". 16,17 Reflecting this perspective, the development of EDPs and its operationalisation into practical guidance has indeed been a process of 'learning by doing', while deepening its theoretical foundation. The overall concept of EDPs stems from the general principle of legitimacy, the definition of four elements is a practical translation of the Accountability for Reasonableness framework, 5 the definition of practical steps is based on existing HTA methods and tools, whereas related recommendations on best practices are inferred from observed practices of HTA bodies around the world. The development process itself was geared through academic knowledge exchange 15,[18][19][20][21][22][23][24][25][26][27][28] and the experience of implementing EDPs in several countries.
We also surveyed HTA bodies and experts around the globe on their need for guidance 26,28 .
The latter activity revealed a strong need for support on most steps of the EDP process. For example, 64% of surveyed INAHTA (International Network of Agencies for Health Technology Assessment) members and 84% of surveyed LMIC experts express a need for guidance on Step D3. Appraisal (Table 1). The guide is organised around the steps of EDPs and addresses a total of 58 questions that HTA bodies may have on how they can best improve the legitimacy of their processes (see Box 1) 29 . The next section summarises the recommendations.
In addition, for each EDP step, we considered the relevance of the four elements of legitimacy and selected a total of 27 key issues of legitimacy based on their importance and data availability. For example, for step A of the EDP process 'Installing an advisory committee', we we only provide a table for step A 'Installing an advisory committee', tables for the other EDP steps can be found in Supplementary file 1: tables S1-S6).

EDPs practical steps: recommendations and HTA practices around the globe
Step A Installing an advisory committee

Role
We advise HTA bodies to install an advisory committee which main task it is to prepare advisory or binding recommendations on the public reimbursement and pricing of health technologies. 4 Our analysis shows that the recommendations made by six HTA bodies are advisory (Brazil, France, Thailand, Canada, Scotland, Australia) and recommendations by two HTA bodies are binding (Germany, UK) ( Table 2). The committee may be involved in other steps of the HTA process such as the selection of decision criteria (step B) or scoping (step D1). In all its tasks, the committee needs to make social and scientific judgements. We recommend HTA bodies to use deliberation to achieve this: it facilitates the judgement process and aims to create a more coherent and mutual understanding of preferences of recommendations among committee members. 30 All analysed HTA bodies use deliberation in their processes.  As the advisory committee informs public funded decision-making, its members should preferably reflect the broad public interest in its recommendations. This means that the composition of the committee should mirror the diversity of social values present in the population. We advise HTA bodies to include 10-15 formal members in the committee which in practice may consists of two types of members. 31 The first type includes members based on their professional or scientific expertise, such as clinicians, public health experts, ethicists, economists, or epidemiologists. The second type includes members based on the interests they represent, such as patient-and/or carer-organisations or industry. Note that these latter members represent the general interests (e.g. of patients and industry) and not specific interests regarding specific health technologies. All formal committee members should have voting power to have a say in the final recommendation of the advisory committee. Our analysis shows that the number of formal committee members varies between 13 (Brazil, Germany) and 29 members (France), with one HTA body not identifying its size (Thailand).

Composition
Each advisory committee includes members of both types, representing expertise and specific interests respectively. Four committees include members representing the public (or consumers) (France, UK, Scotland, Australia) and two include patient representatives (France, Canada). In each committee formal members have voting power ( Table 2).

Selection of committee members
The process for identifying and selecting committee members is preferably done through a transparent approach. The advisory committee should be effectively independent and be free from undue influences. To be cognisant of this and to reduce the risk of undue influence, it is important that committee members sign a conflict-of-interest form before taking on their term and before every meeting. 32

13
Step B Selecting decision criteria Decision criteria reflect the broad goals of a country's health system (such as maximisation of population health, fair distribution of health and financial protection) and underlying values (such as equity, solidarity and access to good quality care). 34 Table S2). Previous research on linking health system values and decision criteria also found that it is difficult to retrieve this information from written sources 35 For learning purposes it would be valuable to study the selection processes of decision criteria as used by HTA bodies in more detail using for example semi-structured interviews.  Table S3).

Step D1 Scoping
Scoping concerns the explicit definition of the objective and research questions of an hta.
Scoping requires the systematic exploration of the relevant aspects of a health technology under evaluation from multiple perspectives (e.g. patients, informal carers, health professionals, decision-makers). Scoping provides important input for the assessment of health technologies in the sense that it defines what evidence needs to be collected. 38 We provide guidance on how to conduct scoping in our guide. We also advise HTA bodies to take up responsibility for scoping, but policy makers, Ministries of Health or external committees, in consultation with relevant stakeholders and/or experts, can also do this.  Table S4).

Step D2 Assessment
The assessment of health technologies includes various activities: systematic evidence collection on the selected decision criteria; synthesising evidence, including quality analysis; independent review of evidence and reporting findings and implications. Our guide does not provide detailed methodological guidance on these activities, as this already exists elsewhere, e.g. in the EUnetHTA core model. 39 Ideally the collection and provision of evidence is carried out by an independent party, such as an academic organisation, to avoid undue influence of any kind. The hta report should be subject to an independent review and discussed by relevant stakeholders, which may lead to revisions, before making the final hta report publicly available.
Our analysis shows that all HTA bodies request an independent review of hta reports; seven HTA bodies consult with stakeholders to inform their assessments, except for Brazil. Also, seven HTA bodies make hta reports publicly available by publishing them on their website, except for Thailand (Supplementary file 1: Table S4).
Step D3 Appraisal

Deliberation
In the appraisal step, the advisory committee interprets the results of the assessment in a broader perspective and formulates a recommendation to inform decision-makers. This is an intrinsically complex and value-laden task and requires a careful judgement process for two  Table S4).

Developing reimbursement decisions
For the sake of legitimacy, reimbursement decisions are ideally be reached by consensus.
However, the development of consensus is not always feasible because stakeholders may, for good reasons, continue to disagree. Also, from a theoretical perspective on legitimacy, it is not always necessary to reach a consensus. The objective of deliberation is to maximise understanding and support among involved stakeholders, realising that not all stakeholders necessarily need to agree with the decision. 5 As such, an advisory committee can also reach a decision by majority voting in case consensus is not achievable. In our analysis two advisory committees rely on consensus as a closure mechanism (Brazil, Thailand), two committees use majority voting if necessary (UK, Australia), and the remaining committees (France, Germany, Canada, Scotland) rely on majority voting by default (Supplementary file 1: Table   S4).

Step E Communication and Appeal
Communication and appeal are important features that enhance the legitimacy of decisionmaking by making the decision and underlying argumentation public, while the conditions of revision and enforcement establish responsiveness and accountability. 5

F Monitoring and Evaluation
Monitoring and evaluation (M&E) concern the process of systematically collecting data over  Table S6).

Discussion
The implementation of EDPs to support health benefit package design in a range of countries in recent years has provided important insights. First, the structured approach of the six EDP steps is holding relevance in a wide variety of settings, e.g. across differences in scope of analysis (from HIV-specific analysis in Indonesia to sector-wide analyses in Pakistan), funding of health technologies (from public funding in Kazakhstan to health insurance funding in Iran) and cultures (from Ghana in the African setting to Ukraine in an east-European setting). Also, the steps are organised in a natural sequence, and this appeared to be a convenient and intuitive order for the planning of activities in these settings. Second, across settings, advisory committee members are well placed to engage in deliberations but often have limited analytical capacity to fully understand the collected evidence. 42  shortcomings and can only be used as a starting point for a deliberative process. 40,43 Analysts should be aware of this, and secure methodological rigour in their approach.
Our analysis of how eight HTA bodies organise key aspects of legitimacy in their HTA processes has several limitations. We provide an overview per March 1, 2021, and countries are continuously updating their processes and the information provided requires to be updated annually. Furthermore, our analysis is based on publicly available sources only, such as HTA bodies' websites. It is well possible that HTA bodies undertake more activities than listed there and included in our overview. Moreover, the publicly available sources do not necessarily specify the quality of their organisational aspects. For example, while HTA bodies may state they communicate the minutes of the advisory committee deliberations, this may in practice simply be state only the decision to include or exclude a health technology without any further argumentation. 44 We consider two main research areas to further support the use of EDPs for health benefit package design. First, as Culyer suggests in his commentary on our earlier publication on EDPs, the best approach to the further development of deliberative processes seems to be "to accumulate the experience of a variety of countries, preferably systematically, from which some general principles might subsequently be inferred". 17  Can I get support to implement EDPs in my country?

Evidence-informed deliberative processes
Why use an evidence-informed deliberative process?
What is an evidence-informed deliberative process?
What are the practical steps?
Why is stakeholder involvement important in EDPs?
What is stakeholder participation?
What is stakeholder consultation?
What is stakeholder communication?

Context
Why is institutional design important and how should it be assessed?
Why is policy context important and how should it be assessed?
What is current HTA capacity and how should it be assessed?
Step A Installing an advisory committee What is the role of an advisory committee?
What should the composition of an advisory committee be?
How can stakeholders get involved in the advisory committee?
How should the members of an advisory committee be identified and selected?
Should an advisory committee be supported by sub-committees?
What is the role of the chair of the advisory committee?
Should the advisory committee use a structured decision-making process?
How should a decision be reached?
How should undue influences in the process be avoided?
Should committee meetings be public?
Should committee members and other stakeholders be trained?
Should committee members be financially compensated?
Step B Selecting decision criteria Why are decision criteria needed?
What are generic decision criteria?
What are contextual decision criteria?

21
How should decision criteria be selected?
Step C. Selecting health technologies for hta What approaches are available for identifying and selecting technologies for hta?
Which approach is best for identifying and selecting technologies for hta?
Which approach should be used for choosing to identify and select health technologies for hta for disinvestment decisions?
Step D1 Scoping What is scoping?
Who should be involved in scoping?
How should scoping be conducted?
Step D2 Assessment What is assessment?
Who does the assessment?
How should hta findings from another setting be adapted?
Step D3 Appraisal What is appraisal?
What is the aim and end product of appraisal?
Should appraisal use an explicit framework to trade-off criteria?
How is deliberation best organised?
How is evidence best presented in the appraisal step?
How much time does the advisory committee need for appraisal?
How can be avoided that an advisory committee says 'yes' to all technologies?
How can the advisory committee trade off the three dimensions of the UHC cube?
How should a decision be reached?
How can all argumentation in an advisory committee be best registered?

Step E Communication and appeal
How should the outcome of the deliberation of the advisory committee be communicated?
How should a formal mechanism for reviewing decisions and addressing disagreements be organised?
Step F Monitoring and evaluation