Evidence-Informed Deliberative Processes for Health Benefit Package Design – Part II: A Practical Guide

Document Type : Original Article


Department for Health Evidence, Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands


Countries around the world are using health technology assessment (HTA) for health benefit package design. Evidence-informed deliberative processes (EDPs) are a practical and stepwise approach to enhance legitimate health benefit package design based on deliberation between stakeholders to identify, reflect and learn about the meaning and importance of values, informed by evidence on these values. This paper reports on the development of practical guidance on EDPs, while the conceptual framework of EDPs is described in a companion paper.

The first guide on EDPs (2019) is further developed based on academic knowledge exchange, surveying 27 HTA bodies and 66 experts around the globe, and the implementation of EDPs in several countries. We present the revised steps of EDPs and how selected HTA bodies (in Australia, Brazil, Canada, France, Germany, Scotland, Thailand and the United Kingdom) organize key issues of legitimacy in their processes. This is based on a review of literature via PubMed and HTA bodies’ websites.

HTA bodies around the globe vary considerable in how they address legitimacy (stakeholder involvement ideally through participation with deliberation; evidence-informed evaluation; transparency; and appeal) in their processes. While there is increased attention for improving legitimacy in decision-making processes, we found that the selected HTA bodies are still lacking or just starting to develop activities in this area. We provide recommendations on how HTA bodies can improve on this.

The design and implementation of EDPs is in its infancy. We call for a systematic analysis of experiences of a variety of countries, from which general principles on EDPs might subsequently be inferred.



Commentary Published on this Paper

  • Reinforcing Science and Policy, With Suggestions for Future Research; Comment on “Evidence-Informed Deliberative Processes for Health Benefit Package Design – Part II: A Practical Guide”

        Abstract | PDF



  1. Chalkidou K, Glassman A, Marten R, et al. Priority-setting for achieving universal health coverage. Bull World Health Organ. 2016;94(6):462-467. doi:2471/blt.15.155721
  2. Verguet S, Hailu A, Eregata GT, Memirie ST, Johansson KA, Norheim OF. Toward universal health coverage in the post-COVID-19 era. Nat Med. 2021;27(3):380-387. doi:1038/s41591-021-01268-y
  3. Glassman A, Giedion U, Smith PC. What's In, What's Out: Designing Benefits for Universal Health Coverage. Washington, DC, United States: Brookings Institution Press, Center for Global Development; 2017.
  4. World Health Organization (WHO). Global Survey on Health Technology Assessment by National Authorities. WHO; 2015.
  5. Daniels N. Accountability for reasonableness. BMJ. 2000;321(7272):1300-1301. doi:1136/bmj.321.7272.1300
  6. World Health Organization (WHO). WHO Consultative Group on Equity and Universal Health Coverage. Making Fair Choices on the Path to UHC. Geneva: WHO; 2016.
  7. Baltussen R, Jansen MP, Bijlmakers L, et al. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256-260. doi:1016/j.jval.2016.11.019
  8. Baltussen R, Jansen M, Oortwijn W. Evidence-informed deliberative processes for health benefit package design – part I: Conceptual framework. Int J Health Policy Manag. doi:10.34172/ijhpm.2021.158
  9. Oortwijn W, Jansen M, Baltussen R. Evidence-Informed Deliberative Process: A Practical Guide for HTA Bodies for Legitimate Benefit Package Design. Nijmegen: Radboud University Medical Center; 2021. Available at: https://www.radboudumc.nl/global-health-priorities.
  10. Oortwijn W, Determann D, Schiffers K, Tan SS, van der Tuin J. Towards integrated health technology assessment for improving decision making in selected countries. Value Health. 2017;20(8):1121-1130. doi:1016/j.jval.2017.03.011
  11. Oortwijn W, Jansen M, Baltussen R. Evidence-informed deliberative processes. Step-by-step practical guide for HTA agencies to enhance legitimate decision-making. Nijmegen, the Netherlands. https://www.radboudumc.nl/global-health-priorities. Published 2019.
  12. Radboud University Medical Center: Global Health Priorities. Country applications of evidence-informed deliberative processes. https://www.radboudumc.nl/global-health-priorities. Accessed March 31, 2021.
  13. Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattananon Y. Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand. Value Health. 2012;15(6):961-970. doi:1016/j.jval.2012.06.006
  14. Zorginstituut Nederland. Pakketadvies in de praktijk: wikken en wegen voor een rechtvaardig pakket. Zorginstituut Nederland; 2017.
  15. Tromp N, Prawiranegara R, Siregar A, et al. Translating international HIV treatment guidelines into local priorities in Indonesia. Trop Med Int Health. 2018;23(3):279-294. doi:1111/tmi.13031
  16. The World Bank. World Development Report 1993: Investing in Health. New York: Oxford University Press; 1993.
  17. Culyer AJ. Use of evidence-informed deliberative processes - learning by doing comment on "use of evidence-informed deliberative processes by health technology assessment agencies around the globe.” Int J Health Policy Manag. 2020;9(6):263-265. doi:15171/ijhpm.2019.116
  18. Baltussen R, Jansen M, Bijlmakers L. Stakeholder participation on the path to universal health coverage: the use of evidence-informed deliberative processes. Trop Med Int Health. 2018;23(10):1071-1074. doi:1111/tmi.13138
  19. Baltussen R, Jansen MP, Bijlmakers L, Tromp N, Yamin AE, Norheim OF. Progressive realisation of universal health coverage: what are the required processes and evidence? BMJ Glob Health. 2017;2(3):e000342. doi:1136/bmjgh-2017-000342
  20. Baltussen R, Mitton C, Danis M, Williams I, Gold M. Global developments in priority setting in health. Int J Health Policy Manag. 2017;6(3):127-128. doi:15171/ijhpm.2017.10
  21. Jansen MP, Baltussen R, Mikkelsen E, et al. Evidence-informed deliberative processes - early dialogue, broad focus and relevance: a response to recent commentaries. Int J Health Policy Manag. 2018;7(1):96-97. doi:15171/ijhpm.2017.88
  22. Jansen MP, Helderman JK, Boer B, Baltussen R. Fair processes for priority setting: putting theory into practice comment on "expanded HTA: enhancing fairness and legitimacy.” Int J Health Policy Manag. 2017;6(1):43-47. doi:15171/ijhpm.2016.85
  23. Jansen MP, Baltussen R, Bærøe K. Stakeholder participation for legitimate priority setting: a checklist. Int J Health Policy Manag. 2018;7(11):973-976. doi:15171/ijhpm.2018.57
  24. Jansen MP, Bijlmakers L, Baltussen R, Rouwette EA, Broekhuizen H. A sustainable approach to universal health coverage. Lancet Glob Health. 2019;7(8):e1013. doi:1016/s2214-109x(19)30252-9
  25. Kapiriri L, Baltussen R, Oortwijn W. Implementing evidence-informed deliberative processes in health technology assessment: a low income country perspective. Int J Technol Assess Health Care. 2020;36(1):29-33. doi:1017/s0266462319003398
  26. Oortwijn W, Jansen M, Baltussen R. Use of evidence-informed deliberative processes by health technology assessment agencies around the globe. Int J Health Policy Manag. 2020;9(1):27-33. doi:15171/ijhpm.2019.72
  27. Seixas BV, Mitton C, Danis M, Williams I, Gold M, Baltussen R. Should priority setting also be concerned about profound socio-economic transformations? a response to recent commentary. Int J Health Policy Manag. 2017;6(12):733-734. doi:15171/ijhpm.2017.85
  28. Oortwijn W, van Oosterhout S, Kapiriri L. Application of evidence-informed deliberative processes in health technology assessment in low- and middle-income countries. Int J Technol Assess Health Care. 2020:1-5. doi:1017/s0266462320000549
  29. Oortwijn W, Jansen M, Baltussen R. Evidence-informed deliberative processes. A practical guide for HTA bodies for legitimate benefit package design. 2nd version. Nijmegen, the Netherlands. https://www.radboudumc.nl/global-health-priorities. Published 2021.
  30. Abelson J, Forest PG, Eyles J, Smith P, Martin E, Gauvin FP. Deliberations about deliberative methods: issues in the design and evaluation of public participation processes. Soc Sci Med. 2003;57(2):239-251. doi:1016/s0277-9536(02)00343-x
  31. Kunz R, Fretheim A, Cluzeau F, et al. Guideline group composition and group processes: article 3 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):229-233. doi:1513/pats.201208-056ST
  32. National Institute for Health and Care Excellence (NICE). Policy on Declaring and Managing Interests for NICE Advisory Committees. London, UK: NICE; 2019.
  33. Abelson J, Wagner F, DeJean D, et al. Public and patient involvement in health technology assessment: a framework for action. Int J Technol Assess Health Care. 2016;32(4):256-264. doi:1017/s0266462316000362
  34. Tromp N, Baltussen R. Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers. BMC Health Serv Res. 2012;12:454. doi:1186/1472-6963-12-454
  35. Oortwijn W, Klein P. Addressing health system values in health technology assessment: the use of evidence-informed deliberative processes. Int J Technol Assess Health Care. 2019;35(2):82-84. doi:1017/s0266462319000187
  36. Henshall C, Oortwijn W, Stevens A, Granados A, Banta D. Priority setting for health technology assessment. Theoretical considerations and practical approaches. Priority setting Subgroup of the EUR-ASSESS Project. Int J Technol Assess Health Care. 1997;13(2):144-185. doi:1017/s0266462300010357
  37. Oortwijn W. on behalf of the Global Policy Forum Facing the dynamics of future innovation: The role of HTA, industry and health system in scanning the horizon -Background paper. https://htai.org/wp-content/uploads/2018/02/HTAi_Global_Policy_Forum_2018_Background_Paper.pdf. Published 2018.
  38. Tanvejsilp P, Ngorsuraches S. Defining the scope of health technology assessment and types of health economic evaluation. J Med Assoc Thai. 2014;97 Suppl 5:S10-16.
  39. European Network on Health Technology Assessment (EUnetHTA). http://www.eunethta.org/. Accessed July 16, 2021. Published 2021.
  40. Baltussen R, Marsh K, Thokala P, et al. Multicriteria decision analysis to support health technology assessment agencies: benefits, limitations, and the way forward. Value Health. 2019;22(11):1283-1288. doi:1016/j.jval.2019.06.014
  41. Pasanen T, Shaxson L. How to Design a Monitoring and Evaluation Framework for a Policy Research Project. London: Overseas Development Institute; 2016.
  42. Baltussen R. Priority Setting in Global Health …. When Reality Beats Science. 2020. Available at: https://decidehealth.world/en/rob.
  43. DiStefano MJ, Krubiner CB. Beyond the numbers: a critique of quantitative multi-criteria decision analysis. Int J Technol Assess Health Care. 2020:1-5. doi:1017/s0266462320000410
  44. Chalkidou K, Li R, Culyer AJ, Glassman A, Hofman KJ, Teerawattananon Y. Health technology assessment: global advocacy and local realities comment on "priority setting for universal health coverage: we need evidence-informed deliberative processes, not just more evidence on cost-effectiveness." Int J Health Policy Manag. 2017;6(4):233-236. doi:15171/ijhpm.2016.118

Articles in Press, Corrected Proof
Available Online from 10 November 2021
  • Receive Date: 09 April 2021
  • Revise Date: 03 November 2021
  • Accept Date: 09 November 2021
  • First Publish Date: 10 November 2021