Challenges and Opportunities for Deliberative Processes for Healthcare Decision-Making; Comment on “Evidence-Informed Deliberative Processes for Health Benefit Package Design – Part II: A Practical Guide”

Introduction

In the last decade, there has been an increasing interest in and discussion about deliberative processes to support healthcare priority setting decisions. A recently published report by an HTAi-ISPOR task force¹ describes a set of minimum features necessary for deliberative processes in health technology assessment (HTA) and provides a checklist and practical recommendations for those seeking to establish, improve or evaluate deliberative processes within HTA. Others have sought to provide practical guidance for deliberative processes by articulating the “supporting actions” that operationalize principles such as transparency, inclusivity, and impartiality.² The recent article by Oortwijn et al on “evidence-informed deliberative processes,”³ (EDPs) and its companion⁴ (also published in this journal) are important additions to the developing body of research and guidance regarding deliberative processes in healthcare priority setting. The guidance in the two papers, which is a helpful summary and clarification of information contained in their Practical Guide for EDPs,⁵ is intended primarily for staff in HTA organizations (at various distances from government) who are responsible for establishing, operating and managing, or monitoring and evaluating EDPs. The aim of the guidance is to provide those staff with examples of and recommendations about how to operationalize the principles of “accountability for reasonableness,” arguably the dominant procedural principles for deliberative processes in HTA globally, with the aim of improving the legitimacy of the decisions of HTA bodies through stakeholder involvement, evidence-informed evaluation, transparency, and mechanisms for appeal. Here I comment briefly on three aspects of Part II of the guidance³ that have important implications for how the guidance is understood and used: viewing current EDPs as “best practice,” conceptualizing the aim of deliberation in terms of social learning among a broad group of stakeholders, and the identification and use of outcomes for monitoring and evaluation.

Viewing Current EDPs as “Best Practice”

It is common within the field of HTA research to label established processes as “best practice,” and, while the authors see their guide as “inspiration” for those looking to improve or establish EDPs, I want to sound a note of caution when inferring recommendations for EDPs from observed practices. Many of the mature or “inspirational” EDPs used as examples have changed over time in response to the organizational, political, economic, and other pressures of their respective health systems. The extent to which these changes have strengthened or weakened the efficiency and legitimacy of priority setting decisions has rarely been examined. As others have noted,⁶ we have a poor understanding currently of how factors other than the evidence being considered influence decisions, and how fair and effective HTA processes can be constructed in light of these potential influences. Accepting current processes as “best practice” without understanding these factors can have important consequences. For example, the EDP guidance recommends that advisory committees have 10-15 members. The sizes of committees vary, and we do not have a good understanding of why different committees have the number of members they do, and how that difference is related, if at all, to the robustness of decision-making, perceptions of fairness, or other desirable outcomes or important principles. This lack of understanding is reflected in the need expressed by HTA agencies for guidance specifically in this area.⁷

Evaluating the legitimacy of established EDPs has fallen primarily to interested and motivated academic researchers. Gathering the documentation that is required to conduct these evaluations relies on close cooperation with HTA bodies, and a degree of transparency that HTA bodies may find challenging. Researchers who have assembled the documentation needed to assess EDPs overseen by England’s National Institute for Health and Care Excellence^8,9 have concluded that changes the organization has made to its procedural and substantive principles has undermined the fairness and legitimacy of its decision-making.⁹ Other HTA bodies have made similar changes. For example, the Canadian Agency for Drugs and Technologies in Health has made changes to its pharmaceutical reimbursement recommendations framework¹⁰; yet, to my knowledge, no assessment has been made of these changes, and there is little information available to explain the overall approach to and rationale for having made those changes.

Many of the recommendations regarding “best practice” are based on EDPs in high-income countries. While this is understandable, Oortwijn and colleagues have gained important insights through their research and experience in working with low- and middle-income countries (LMICs). Given the desire on the part of HTA agencies to find efficiencies in deliberative processes, I would like to highlight the possibility of high-income countries looking to LMICs for inspiration — a process sometimes referred to as “reverse innovation.”¹¹ The imperative to do more with less in LMICs, which may be particularly acute in healthcare, and the absence of established organizational structures and legislation that may constrain novel approaches, could provide an environment within which to find new possibilities for EDPs.

The Aim of Stakeholder Involvement

EDPs enhance the legitimacy of decisions through a deliberation among stakeholders to “identify, reflect and learn about the meaning and importance of values, informed by evidence about these values,” and improving the quality of decision-making by making it better informed,⁴ a process sometimes referred to as “social learning.” Stakeholders are defined broadly to include patients, the broader public, healthcare providers, purchasers, payers, policy makers, product manufacturers, and academic researchers. There are a number of tensions inherent in making the aim of deliberative processes social learning among this broad group of stakeholders.

One point of tension is that this robust and broad ranging social learning may not be needed to secure the legitimacy of all decisions. EDPs view deliberation as a means to strengthening the legitimacy of decisions. While this learning may be important when making decisions about highly contested conditions or technologies, the more typical decisions of HTA bodies may neither require nor need this kind of interaction among stakeholders to satisfy sufficiently the conditions of accountability for reasonableness, which has as its focus the development and presentation of facts, reasons, and principles that would be found reasonable to those affected by the decision.This is a point about determining what is sufficient for legitimacy and not about settling controversy. If the careful, sensitive, and robust deliberation that constitutes social learning is not always required, then taking social learning as the main aim of deliberation (as a means to strengthen legitimacy) may result in unnecessary time and resource use for health benefit package decisions. There are processes and techniques falling between consultation and participation, such as Delphi, that may help to address these challenges.¹²

HTA agencies, generally speaking, are not striving to reach an ideal process to secure legitimacy, but seeking to find efficiencies and sufficiency in addressing the various perspectives. Individual HTA organizations will need to find the best balance of procedural principles and practical efficiencies for their context when constructing EDPs. Hence, there are two practical questions that might be helpfully answered by the EDP guidance: (1) How might we determine which decisions could benefit most from or require a deepened engagement oriented toward social learning to further the aims of legitimacy and when might other less robust forms of involvement be used? (2) What are the relevant knowledge and skills required to effectively support values-based conversations of this kind, and to reveal and handle the value conflicts and moral uncertainty that may arise?

Given the location of EDPs at the intersection of democratic governance and the application of research-based knowledge, some users of the guidance might reasonably conclude from the authors’ description of stakeholder involvement that deliberative processes function like a legislature, with the corollary that the aim of deliberation is to settle value conflicts in a way that would be satisfactory to each stakeholder.¹³ If this were to happen, there is a subsequent risk that the activity of deliberation is organized as a negotiation of interests and the ethical challenges of benefit package decisions are transformed into challenges that are resolved politically. It should be remembered that, for those involved in EDPs, there is an obligation on the part of all those who contribute to and benefit from a health system based on shared savings to restrain their demands in order to maintain the viability of that system.¹³

Further Developing Monitoring and Evaluation

An evaluation of EDPs “…ideally informs the HTA body about any shortcomings in terms of how their processes are being implemented and/or its overall impact and why this may be so.”³ Despite its importance, evaluation and monitoring are not well described by HTA agencies.³ As HTA bodies strive to do better on this front, I want to highlight some potential challenges in evaluation that are raised by the complex social, political, and ethical aims of EDPs, and that ought to be taken up by the HTA community.

Not mentioned by the authors in their article, but described by them in their guide, is the importance of developing a theory of change to “…explain how the implementation of an EDP is expected to achieve desired impacts, in terms of inputs and activities, outputs, outcomes, and more long-term impacts.”⁵ This is an important point, and seems much neglected by HTA bodies; those developing and evaluating EDPs would do well to accept this challenge. As was noted earlier, in their efforts to improve EDPs in terms of more proximate process outcomes, such as consistency in decision-making, timeliness, greater acceptability of decisions, and speedier implementation of or access to new technologies, well-established EDPs, may have become less transparent and accountable, thus weakening legitimacy. A well-developed theory of change may help to identify proximate actions that HTA bodies can take to improve practice, while avoiding moving in directions that undermine their desired goals. This point is especially important because techniques, such as multi-criteria decision analysis, which are intended to incorporate a broad range of values and improve consistency in decision-making, require trade-offs of this kind.

According to the authors, the main aims of EDPs are securing or improving the reasonableness of decisions as perceived by stakeholders, and sharing perspectives and values to maximize understanding and support among stakeholders. How might this “reasonableness” and “maximized understanding” be assessed? Researchers have noted that there is a need to evaluate the reasonableness of both procedural and substantive principles within priority setting processes.¹³ Few, if any, HTA bodies provide sufficiently transparent and detailed records of deliberations to be able to monitor and assess the development of “coherence and mutual understanding” among committee members. What is the appropriate standard for this coherence and mutual understanding? For example, many committees use a consensus model for decision-making. While many committees describe this aspect of their decision-making in their terms of reference (or similar documentation), I am not aware of an HTA body that has provided an explicit rationale for the use of consensus or connected this model explicitly to the aims of EDPs, as has been attempted for ethics committees, for example.¹⁴ Better articulating the nature of the committees and their decisions will have important implications for how committees operate, such as the number of members and their expertise.

HTA bodies must balance various competing values and actions to provide recommendations or decisions at the appropriate time for decision makers. To do so, gains on one feature of EDPs, for example, timeliness of outputs, may come at the expense of another, such as stakeholder involvement in input and deliberation. There are opportunity costs for structuring EDPs in one way rather than another, and not all agencies will choose to make the same trade-offs. Evaluations must be conducted carefully, and in detail, so as not to oversimply the complex considerations and competing values that have led to an EDP being set up in a particular way. In EDPs, as with many other things, context and details are important. As one of the guide’s authors has elsewhere concluded “…best practices that may work well in some countries might not necessarily work evenly well in other countries.”¹⁵

Conclusion

The updated practical guide on EDPs provides helpful examples and recommendations for those seeking to establish or improve deliberative processes to support the legitimacy of health benefit package decisions. In light of our current understanding of EDPs, I have pointed to some reasons for caution when looking to existing processes for inspiration, to some challenges with conceiving of deliberation in terms of social learning, and when identifying and selecting outcomes for evaluating and comparing EDPs. Nevertheless, it is important to acknowledge the complexity of ensuring legitimate decision-making and be respectful of the time, effort, and resources governments and HTA bodies devote to establishing and managing robust and meaningful EDPs, and to not be overly critical. Exercising appropriate caution, current processes can, indeed, inspire.

Acknowledgements

I would like to thank Professor Julia Abelson (McMaster University, ON, Canada) for helpful comments on an early draft of this commentary. I would also like to thank an anonymous reviewer for their thoughtful and constructive comments.

Ethical issues

Not applicable.

Competing interests

Author declares that he has no competing interests.

Author’s contribution

KB is the single author of the paper.

References

1. Oortwijn W, Husereau D, Abelson J. Designing and implementing deliberative processes for health technology assessment: a good practices report of a joint HTAi/ISPOR task force. Int J Technol Assess Health Care 2022; 38(1):e37. doi: 10.1017/s0266462322000198 [Crossref] [ Google Scholar]
2. Bond K, Stiffell R, Ollendorf DA. Principles for deliberative processes in health technology assessment. Int J Technol Assess Health Care. 2020:1-8. 10.1017/s0266462320000550.
3. Oortwijn W, Jansen M, Baltussen R. Evidence-informed deliberative processes for health benefit package design - part II: a practical guide. Int J Health Policy Manag 2022; 11(10):2327-2336. doi: 10.34172/ijhpm.2021.159 [Crossref] [ Google Scholar]
4. Baltussen R, Jansen M, Oortwijn W. Evidence-informed deliberative processes for legitimate health benefit package design - part I: conceptual framework. Int J Health Policy Manag. 2021. 10.34172/ijhpm.2021.158.
5. Oortwijn W, Jansen M, Baltussen R. Evidence-informed deliberative processes. A practical guide for HTA bodies for legitimate benefit package design. Version 2.0. Nijmegen: Radboud University Medical Center; 2021. https://www.radboudumc.nl/getmedia/17a96fdb-553b-4e68-81ab-4d8d9a7f9ff1/UMCRadboud_Guide_17x24_inside_DEF_WEB.aspx. Accessed June 11, 2022.
6. Ollendorf D, Krubiner C. The dynamics of health technology assessment: Is it just about the evidence? Centre for Global Development Blog. Washington, DC: Centre for Global Development. https://www.cgdev.org/blog/dynamics-health-technology-assessment-it-just-about-evidence. Updated November 5, 2019. Accessed June 9, 2022.
7. Oortwijn W, Jansen M, Baltussen R. Use of evidence-informed deliberative processes by health technology assessment agencies around the globe. Int J Health Policy Manag 2020; 9(1):27-33. doi: 10.15171/ijhpm.2019.72 [Crossref] [ Google Scholar]
8. Littlejohns P, Chalkidou K, Culyer AJ. National Institute for Health and Care Excellence, social values and healthcare priority setting. J R Soc Med 2019; 112(5):173-179. doi: 10.1177/0141076819842846 [Crossref] [ Google Scholar]
9. Charlton V. NICE and fair? Health technology assessment policy under the UK’s National Institute for Health and Care Excellence, 1999-2018. Health Care Anal 2020; 28(3):193-227. doi: 10.1007/s10728-019-00381-x [Crossref] [ Google Scholar]
10. Canadian Agency for Drugs and Technologies in Health (CADTH). Common drug review reimbursement recommendations options and deliberative framework — effective November 21, 2012, CDR update – issue 83. In: CADTH drug program updates archive: May 2003-July 2020. https://www.cadth.ca/sites/default/files/cdr/CADTH-Archived-Updates.pdf. Accessed May 15, 2022.
11. Bhattacharyya O, Wu D, Mossman K. Criteria to assess potential reverse innovations: opportunities for shared learning between high- and low-income countries. Global Health 2017; 13(1):4. doi: 10.1186/s12992-016-0225-1 [Crossref] [ Google Scholar]
12. Cantrill JA, Sibbald B, Buetow S. The Delphi and nominal group techniques in health services research. Int J Pharm Pract 1996; 4(2):67-74. doi: 10.1111/j.2042-7174.1996.tb00844.x [Crossref] [ Google Scholar]
13. Rumbold B, Weale A, Rid A, Wilson J, Littlejohns P. Public reasoning and health-care priority setting: the case of NICE. Kennedy Inst Ethics J 2017; 27(1):107-134. doi: 10.1353/ken.2017.0005 [Crossref] [ Google Scholar]
14. Moreno JD. Ethics by committee: the moral authority of consensus. J Med Philos 1988; 13(4):411-432. doi: 10.1093/jmp/13.4.411 [Crossref] [ Google Scholar]
15. Oortwijn W, Determann D, Schiffers K, Tan SS, van der Tuin J. Towards integrated health technology assessment for improving decision making in selected countries. Value Health 2017; 20(8):1121-1130. doi: 10.1016/j.jval.2017.03.011 [Crossref] [ Google Scholar]