Use of Evidence-informed Deliberative Processes by Health Technology Assessment Agencies Around The Globe

Document Type: Original Article


Department for Health Evidence, Radboud Institute for Health Sciences, Radboud University Medical Centre, Nijmegen, The Netherlands


Evidence-informed deliberative processes (EDPs) were recently introduced to guide health technology assessment (HTA) agencies to improve their processes towards more legitimate decision-making. The EDP framework provides guidance that covers the HTA process, ie, contextual factors, installation of an appraisal committee, selecting health technologies and criteria, assessment, appraisal, and communication and appeal. The aims of this study were to identify the level of use of EDPs by HTA agencies, identify their needs for guidance, and to learn about best practices.
A questionnaire for an online survey was developed based on the EDP framework, consisting of elements that reflect each part of the framework. The survey was sent to members of the International Network of Agencies for Health Technology Assessment (INAHTA). Two weeks following the invitation, a reminder was sent. The data collection took place between September-December 2018.
Contact persons from 27 member agencies filled out the survey (response rate: 54%), of which 25 completed all questions. We found that contextual factors to support HTA development and the critical elements regarding conducting and reporting on HTA are overall in place. Respondents indicated that guidance was needed for specific elements related to selecting technologies and criteria, appraisal, and communication and appeal. With regard to best practices, the Canadian Agency for Drugs and Technologies and the National Institute for Health and Care Excellence (NICE, UK) were most often mentioned.
This is the first survey among HTA agencies regarding the use of EDPs and provides useful information for further developing a practical guide for HTA agencies around the globe. The results could support HTA agencies in improving their processes towards more legitimate decision-making, as they could serve as a baseline measurement for future monitoring and evaluation.



Supplementary File 1 (Download)


Commentaries Published on this Paper


  • Use of Evidence-Informed Deliberative Processes – Learning by Doing; Comment on “Use of Evidence-informed Deliberative Processes by Health Technology Assessment Agencies Around the Globe”

          Abstract | PDF


  • HTA Agencies Need Evidence-Informed Deliberative Processes; Comment on “Use of Evidence-Informed Deliberative Processes by Health Technology Assessment Agencies Around the Globe”

          Abstract | PDF


  • Deliberative Processes by Health Technology Assessment Agencies: A Reflection on Legitimacy, Values and Patient and Public Involvement; Comment on “Use of Evidence-informed Deliberative Processes by Health Technology Assessment Agencies Around the Globe”

         Abstract | PDF



Main Subjects

  1. Børlum Kristensen F, Husereau D, Huić M, et al. Identifying the need for good practices in health technology assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA. Value in Health. 2019;22(1):13-20. doi:10.1016/j.jval.2018.08.010
  2. Baltussen R, Jansen PMJ, Bijlmakers L, et al. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256-260. doi:10.1016/j.jval.2016.11.019.v
  3. Oortwijn W, Determann D, Schiffers K, et al. Towards integrated health technology assessment for improving decision-making in selected countries. Value Health. 2017;20(8):1121-1130. doi:10.1016/j.jval.2017.03.011
  4. Abelson J, Forest PG, Eyles J, et al. Deliberations about Deliberative Methods: Issues in the design and evaluation of public participation processes. Soc Sci Med. 2003;57(2):239-251. doi:10.1016/s0277-9536(02)00343-x
  5. Rawlins MD, Culyer AJ. National Institute for Clinical Excellence and its value judgments BMJ. 2004;329:224. doi:10.1136/bmj.329.7459.224
  6. Lomas J, Culyer AJ, McCutcheon C, et al. Conceptualizing and Combining Evidence for Health System Guidance. Ottawa: Canadian Health Services Research Foundation; 2005.
  7. Culyer AJ, Lomas J. Deliberative processes and evidence-informed decision making in healthcare: do they work and how might we know? Evid Policy. 2006;2(3):357-371.  doi:10.1332/174426406778023658
  8. Culyer AJ. Deliberative Processes in Decisions about Health Care Technologies: Combining Different Types of Evidence, Values, Algorithms and People, London: Office of Health Economics; 2009.
  9. Dobrow MJ, Chafe R, Burchett HED, et al. Designing deliberative methods for combining heterogeneous evidence: a systematic review and qualitative scan. Research conducted: 2009. Ottawa: Canadian Foundation for Healthcare Improvement; 2012.
  10. Baltussen R, Jansen M, Bijlmakers L. Stakeholder participation on the path to universal health coverage: the use of evidence-informed deliberative processes. Trop Med Int Health. 2018;23(10):1071-1074. doi:10.1111/tmi.13138
  11. Chalkidou K, Li R, Culyer T, et al. Health technology assessment: global advocacy and local realities; Comment on “Priority setting for universal health coverage: we need evidence-informed deliberative processes, not just more evidence on cost-effectiveness.” Int J Health Policy Manag. 2017;6(4):233-236. doi:10.15171/ijhpm.2016.118
  12. Oortwijn W, Jansen M, Baltussen R. Evidence-informed deliberative processes. A practical guide for HTA agencies to enhance legitimate decision-making. Version 1.0. Nijmegen: Radboud University Medical Centre, Radboud Institute for Health Sciences; 2019.   
  13. INAHTA. A checklist for health technology assessment reports. INAHTA Secretariat, Version 3.2. August 2007.  
  14. Hailey D, Werkö S, Bakri R, et al. Involvement of consumers in health technology assessment activities by INAHTA agencies. Int J Technol Assess Health Care. 2013; 29(1):79-83. doi:10.1017/S026646231200075X
  15. Merlin T, Tamblyn D, Ellery B. What’s in a name? Developing definitions for common health technology assessment product types of the International Network of Agencies for Health Technology Assessment (INAHTA). Int J Technol Assess Health Care. 2014;30(4):430-437. doi:10.1017/S0266462314000543
  16. EUnetHTA. EUnetHTA WP7 research and analysis activity 1. An analysis of HTA and reimbursement procedures in EUnetHTA partner countries: Annex 2 Case studies, p. 4-8.  
  17. About EuroScan. EuroScan Website.  
  18. EUnetHTA Joint Action 2, Work Package 8. HTA Core Model ® version 3.0.  Published 2016.
  19. Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration; 2011.
  20. Jansen MPM, Baltussen R, Bærøe K. Stakeholder Participation for Legitimate Priority Setting: A Checklist. Int J Health Policy Manag. 2018;7(11):973-976. doi: 10.15171/ijhpm.2018.57
  21. Baltussen R, Thokala P, Marsh K, et al. Multi criteria decision analysis to support HTA agencies – benefits, limitations and the way forward. Value in Health. 2019; In Press.
  22. Chalkidou K, Glassman A, Marten R, et al. Priority-setting for achieving universal health coverage. Bull World Health Organ. 2016;94:462-467. doi:10.2471/BLT.15.155721
  23. Glassman A, Giedion U, Sakuma Y, et al. Defining a health benefits package: what are the necessary processes? Health Syst Reform. 2016;2(1):39-50. doi:10.1080/23288604.201 6.1124171
  24. Pichon-Riviere A, Soto N, Augustovski F, et al. Health technology assessment for decision making in Latin America: good practice principles. Int J Technol Assess Health Care. 2018;34(3):241-247. doi:10.1017/S0266462318000326
  25. Pichon-Riviere A, Soto N, Augustovski F, et al. Stakeholder involvement in the health technology assessment process in Latin America. Int J Technol Assess Health Care. 2018;34(3):248-253. doi:10.1017/S0266462318000302
  26. Mundy L, Trowman R, Kearney B. Universal Healthcare in Asia: HTA and Real-World Data Overcoming Barriers. DIA Newsletter. January 2018.  
  27. Neumann P, Drummond M, Jönsson B, et al. Are Key Principles for improved health technology assessment supported and used by health technology assessment organizations? Int J Technol Assess Health Care. 2010;26(1):71-78. doi:10.1017/S0266462309990833