Moving Towards Accountability for Reasonableness – A Systematic Exploration of the Features of Legitimate Healthcare Coverage Decision-Making Processes Using Rare Diseases and Regenerative Therapies as a Case Study

Document Type: Original Article


1 Analytica Laser, Montreal, QC, Canada

2 Analytica Laser, London, UK

3 Analytica Laser, New York City, NY, USA

4 National Health Care Institute (ZIN), Diemen, The Netherlands

5 Pfizer Inc, New York City, NY, USA (retired)

6 Liège University, Liège, Belgium

7 Center for Public Health Research, University of Milan-Bicocca, Milan, Italy

8 National School of Public Health IMIENS-UNED, Madrid, Spain

9 Department of Public Health, University of Alcalá, Alcalá de Henares, Spain

10 University of Leeds, Leeds, UK

11 Division of Health Economics, German Cancer Research Center (DKFZ), Heidelberg, Germany

12 University of Heidelberg, Heidelberg, Germany

13 ASST Niguarda and Regione Lombardia, Welfare Directorate, Milano, Italy


The accountability for reasonableness (A4R) framework defines 4 conditions for legitimate healthcare coverage decision processes: Relevance, Publicity, Appeals, and Enforcement. The aim of this study was to reflect on how the diverse features of decision-making processes can be aligned with A4R conditions to guide decision-making towards legitimacy. Rare disease and regenerative therapies (RDRTs) pose special decision-making challenges and offer therefore a useful case study.

Features operationalizing each A4R condition as well as three different approaches to address these features (cost-per-QALY-focused and multicriteria-based) were defined and organized into a matrix. Seven experts explored these features during a panel run under the Chatham House Rule and provided general and RDRT-specific recommendations. Responses were analyzed to identify converging and diverging recommendations.

Regarding Relevance, recommendations included supporting deliberation, stakeholder participation and grounding coverage decision criteria in normative and societal objectives. Thirteen of 17 proposed decision criteria were recommended by a majority of panelists. The usefulness of universal cost-effectiveness thresholds to inform allocative efficiency was challenged, particularly in the RDRT context. RDRTs raise specific issues that need to be considered; however, rarity should be viewed in relation to other aspects, such as disease severity and budget impact. Regarding Publicity, panelists recommended transparency about the values underlying a decision and value judgements used in selecting evidence. For Appeals, recommendations included a life-cycle approach with clear provisions for re-evaluations. For Enforcement, external quality reviews of decisions were recommended.

Moving coverage decision-making processes towards enhanced legitimacy in general and in the RDRT context involves designing and refining approaches to support participation and deliberation, enhancing transparency, and allowing explicit consideration of multiple decision criteria that reflect normative and societal objectives.


Supplementray File 1 (Download)


Main Subjects

  1. Definition of legitimacy in English by Oxford Dictionaries. English Oxford Dictionary website.  Updated 2018.
  2. Terwindt F, Rajan D, Soucat A. Priority-setting for national health policies, strategies and plans. In: Strategizing national health in the 21st century: a handbook. World Health Organization; 2016.
  3. Clark S, Weale A. Social values in health priority setting: a conceptual framework. J Health Organ Manag. 2012;26(3):293-316. doi:10.1108/14777261211238954
  4. Daniels N. Justice, health, and healthcare. Am J Bioeth. 2001;1(2):2-16. doi:10.1162/152651601300168834
  5. Goetghebeur M, Castro-Jaramillo H, Baltussen R, Daniels N. The art of priority setting. Lancet. 2017;389(10087):2368-2369. doi:10.1016/s0140-6736(17)31573-8
  6. Daniels N. Decisions about access to health care and accountability for reasonableness. J Urban Health. 1999;76(2):176-191. doi:10.1007/bf02344674
  7. Daniels N. Accountability for reasonableness. BMJ. 2000;321(7272):1300-1301.
  8. Daniels N, van der Wilt GJ. Health technology assessment, deliberative process, and ethically contested issues. Int J Technol Assess Health Care. 2016;32(1-2):10-15. doi:10.1017/s0266462316000155
  9. Daniels N, Sabin J. Limits to health care: fair procedures, democratic deliberation, and the legitimacy problem for insurers. Philos Public Aff. 1997;26(4):303-350.
  10. Hasman A, Holm S. Accountability for reasonableness: opening the black box of process. Health Care Anal. 2005;13(4):261-273. doi:10.1007/s10728-005-8124-2
  11. Badano G. If You're a Rawlsian, How Come You're So Close to Utilitarianism and Intuitionism? A Critique of Daniels's Accountability for Reasonableness. Health Care Anal. 2018;26(1):1-16. doi:10.1007/s10728-017-0343-9
  12. Friedman A. Beyond accountability for reasonableness. Bioethics. 2008;22(2):101-112. doi:10.1111/j.1467-8519.2007.00605.x
  13. Daniels N, Porteny T, Urritia J. Expanded HTA: enhancing fairness and legitimacy. Int J Health Policy Manag. 2015;5(1):1-3. doi:10.15171/ijhpm.2015.187
  14. Baltussen R, Jansen MPM, Bijlmakers L, et al. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256-260. doi:10.1016/j.jval.2016.11.019
  15. Culyer AJ. Ethics, priorities and cancer. J Cancer Policy. 2017;11:6-11. doi:10.1016/j.jcpo.2016.09.007
  16. National Institute for Health and Care Excellence. NICE Process [PMG9]. Guide to the methods of technology appraisal 2013. National Institute for Health and Care Excellence website.  Updated 2013. Accessed August 14, 2016.
  17. Gold MR, Stevenson D, Fryback DG. HALYS and QALYS and DALYS, Oh My: similarities and differences in summary measures of population Health. Annu Rev Public Health. 2002;23:115-134. doi:10.1146/annurev.publhealth.23.100901.140513
  18. Angelis A, Kanavos P. Multiple Criteria Decision Analysis (MCDA) for evaluating new medicines in Health Technology Assessment and beyond: The Advance Value Framework. Soc Sci Med. 2017;188:137-156. doi:10.1016/j.socscimed.2017.06.024
  19. Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Evidence and Value: Impact on DEcisionMaking--the EVIDEM framework and potential applications. BMC Health Serv Res. 2008;8:270. doi:10.1186/1472-6963-8-270
  20. Radaelli G, Lettieri E, Masella C, Merlino L, Strada A, Tringali M. Implementation of EUnetHTA core Model(R) in Lombardia: the VTS framework. Int J Technol Assess Health Care. 2014;30(1):105-112. doi:10.1017/s0266462313000639
  21. Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattananon Y. Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand. Value Health. 2012;15(6):961-970. doi:10.1016/j.jval.2012.06.006
  22. Regier DA, Peacock S. Theoretical foundations of MCDA. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, eds. Multi-Criteria Decision Analysis to Support Healthcare Decisions. Switzerland: Springer International Publishing; 2017:9-28.
  23. Paulden M, Stafinski T, Menon D, McCabe C. Value-based reimbursement decisions for orphan drugs: a scoping review and decision framework. Pharmacoeconomics. 2015;33(3):255-269. doi:10.1007/s40273-014-0235-x
  24. Phelps CE, Madhavan G. Using multicriteria approaches to assess the value of health care. Value Health. 2017;20(2):251-255. doi:10.1016/j.jval.2016.11.011
  25. Wagner M, Khoury H, Willet J, Rindress D, Goetghebeur M. Can the EVIDEM framework tackle issues raised by evaluating treatments for rare diseases: analysis of issues and policies, and context-specific adaptation. Pharmacoeconomics. 2016;34(3):285-301. doi:10.1007/s40273-015-0340-5
  26. Mitton CR, McMahon M, Morgan S, Gibson J. Centralized drug review processes: are they fair? Soc Sci Med. 2006;63(1):200-211. doi:10.1016/j.socscimed.2005.11.049
  27. Jansson S. Implementing accountability for reasonableness--the case of pharmaceutical reimbursement in Sweden. Health Econ Policy Law. 2007;2(Pt 2):153-171. doi:10.1017/s1744133107004082
  28. Schlander M. NICE accountability for reasonableness: a qualitative study of its appraisal of treatments for attention-deficit/hyperactivity disorder (ADHD). Curr Med Res Opin. 2007;23(1):207-222. doi:10.1185/030079906x159461
  29. Schlander M. The use of cost-effectiveness by the National Institute for Health and Clinical Excellence (NICE): no (t yet an) exemplar of a deliberative process. J Med Ethics. 2008;34(7):534-539. doi:10.1136/jme.2007.021683
  30. Greenberg D, Siebzehner MI, Pliskin JS. The process of updating the National List of Health Services in Israel: is it legitimate? Is it fair? Int J Technol Assess Health Care. 2009;25(3):255-261. doi:10.1017/s026646230999016x
  31. Goetghebeur M, Wagner M. Identifying value (s): a reflection on the ethical aspects of MCDA in healthcare decisionmaking. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, eds. Multi-Criteria Decision Analysis to Support Healthcare Decisions. Switzerland: Springer International Publishing; 2017:29-46.
  32. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood). 2008;27(3):759-769. doi:10.1377/hlthaff.27.3.759
  33. Drummond M, Towse A. Orphan drugs policies: a suitable case for treatment. Eur J Health Econ. 2014;15(4):335-340. doi:10.1007/s10198-014-0560-1
  34. Gericke CA, Riesberg A, Busse R. Ethical issues in funding orphan drug research and development. J Med Ethics. 2005;31(3):164-168. doi:10.1136/jme.2003.007138
  35. Mahalatchimy A. Reimbursement of cell-based regenerative therapy in the UK and France. Med Law Rev. 2016;24(2):234-258. doi:10.1093/medlaw/fww009
  36. Corbett MS, Webster A, Hawkins R, Woolacott N. Innovative regenerative medicines in the EU: a better future in evidence? BMC Med. 2017;15(1):49. doi:10.1186/s12916-017-0818-4
  37. Nicod E. Why do health technology assessment coverage recommendations for the same drugs differ across settings? Applying a mixed methods framework to systematically compare orphan drug decisions in four European countries. Eur J Health Econ. 2017;18(6):715-730. doi:10.1007/s10198-016-0823-0
  38. Daniels N, Sabin EJ.  Setting limits fairly: Learning to Share Resources for Health. 2nd ed. New York: Oxford University Press; 2008.
  39. Sandman L, Gustavsson E. Beyond the Black Box Approach to Ethics! Comment on "Expanded HTA: Enhancing Fairness and Legitimacy." Int J Health Policy Manag. 2016;5(6):393-394. doi:10.15171/ijhpm.2016.43
  40. Abrishami P, Oortwijn W, Hofmann B. Ethics in HTA: Examining the "Need for Expansion." Int J Health Policy Manag. 2017;6(10):551-553. doi:10.15171/ijhpm.2017.43
  41. Culyer AJ. HTA - Algorithm or Process? Comment on "Expanded HTA: Enhancing Fairness and Legitimacy." Int J Health Policy Manag. 2016;5(8):501-505. doi:10.15171/ijhpm.2016.59
  42. Oortwijn W, Determann D, Schiffers K, Tan SS, van der Tuin J. Towards integrated health technology assessment for improving decision making in selected countries. Value Health. 2017;20(8):1121-1130. doi:10.1016/j.jval.2017.03.011
  43. European Commission. Process on Corporate Social Responsibility in the Field of Pharmaceuticals Platform on Access to Medicines in Europe Working Group on Mechanism of Coordinated Access to Orphan Medicinal Products (MoCA-OMP). Transparent Value Framework; 2014.
  44. Hughes-Wilson W, Palma A, Schuurman A, Simoens S. Paying for the Orphan Drug System: break or bend? Is it time for a new evaluation system for payers in Europe to take account of new rare disease treatments? Orphanet J Rare Dis. 2012;7:74. doi:10.1186/1750-1172-7-74
  45. Iskrov G, Miteva-Katrandzhieva T, Stefanov R. Multi-criteria decision analysis for assessment and appraisal of orphan drugs. Front Public Health. 2016;4:214. doi:10.3389/fpubh.2016.00214
  46. Kolasa K, Zwolinski KM, Kalo Z, Hermanowski T. Potential impact of the implementation of multiple-criteria decision analysis (MCDA) on the Polish pricing and reimbursement process of orphan drugs. Orphanet J Rare Dis. 2016;11:23. doi:10.1186/s13023-016-0388-0
  47. Sussex J, Rollet P, Garau M, Schmitt C, Kent A, Hutchings A. A pilot study of multicriteria decision analysis for valuing orphan medicines. Value Health. 2013;16(8):1163-1169. doi:10.1016/j.jval.2013.10.002
  48. Schlander M, Garattini S, Holm S, et al. Incremental cost per quality-adjusted life year gained? The need for alternative methods to evaluate medical interventions for ultra-rare disorders. J Comp Eff Res. 2014;3(4):399-422. doi:10.2217/cer.14.34
  49. Nord E. Beyond QALYs: Multi-criteria based estimation of maximum willingness to pay for health technologies. Eur J Health Econ. 2018;19(2):267-275. doi:10.1007/s10198-017-0882-x
  50. National Institute for Health and Care Excellence (NICE). Interim process and methods of the highly specialised technologies programme. Updated to reflect 2017 changes. National Institute for Health and Care Excellence website.  Updated 2017. Accessed May 24, 2017.
  51. National Institute for Health and Care Excellence (NICE). NICE and NHS England consultation on changes to the arrangements for evaluating and funding drugs and other health technologies assessed through NICE's technology appraisal and highly specialised technologies programmes. National Institute for Health and Care Excellence website.  Updated 2017. Accessed May 24, 2017.
  52. Scottish Medicines Consortium (SMC). SMC modifiers used in appraising new medicines. SMC website.  Updated 2017. Accessed April 13, 2017.
  53. Scottish Medicines Consortium (SMC). PACE (Patient & Clinician Engagement) overview document. SMC website.   Updated 2016. Accessed May 10, 2017.
  54. Hettle R, Corbett M, Hinde S, et al. The assessment and appraisal of regenerative medicines and cell therapy products: an exploration of methods for review, economic evaluation and appraisal. Health Technol Assess. 2017;21(7):1-204. doi:10.3310/hta21070
  55. Thokala P, Devlin N, Marsh K, et al. Multiple criteria decision analysis for health care decision making--an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(1):1-13. doi:10.1016/j.jval.2015.12.003
  56. Marsh K, M IJ, Thokala P, et al. Multiple criteria decision analysis for health care decision making--emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(2):125-137. doi:10.1016/j.jval.2015.12.016
  57. Frenk J. The global health system: strengthening national health systems as the next step for global progress. PLoS Med. 2010;7(1):e1000089. doi:10.1371/journal.pmed.1000089
  58. Ronfard V, Vertes AA, May MH, Dupraz A, van Dyke ME, Bayon Y. Evaluating the past, present, and future of regenerative medicine: a global view. Tissue Eng Part B Rev. 2017;23(2):199-210. doi:10.1089/ten.TEB.2016.0291
  59. Buckland KF, Bobby Gaspar H. Gene and cell therapy for children--new medicines, new challenges? Adv Drug Deliv Rev. 2014;73:162-169. doi:10.1016/j.addr.2014.02.010
  60. Royal Institute of International Affairs. Chatham House Rule. Royal Institute of International Affairs website.  Updated 2002. Accessed September 15, 2015.
  61. Schlander M, Telser H, Holm S, Marshall D, Nord E, Richardson J. The European Social Preference Measurement (ESPM) study: Conceptual considerations and implementation. Poster presented at: ISPOR 19th Annual European Congress; Oct 29-Nov 2, 2016; Vienna, Austria.
  62. Tanios N, Wagner M, Tony M, et al. Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. Int J Technol Assess Health Care. 2013;29(4):456-465. doi:10.1017/s0266462313000573
  63. Lakdawalla DN, Doshi JA, Garrison LP Jr, Phelps CE, Basu A, Danzon PM. Defining elements of value in health care--a health economics approach: an ISPOR Special Task Force report [3]. Value Health. 2018;21(2):131-139. doi:10.1016/j.jval.2017.12.007
  64. Annemans L, Ayme S, Le Cam Y, et al. Recommendations from the European working Group for Value Assessment and Funding Processes in rare diseases (ORPH-VAL). Orphanet J Rare Dis. 2017;12(1):50. doi:10.1186/s13023-017-0601-9
  65. EURORDIS. Breaking the access deadlock to leave no one behind. EURORDIS website.  Updated 2018. Accessed December 18, 2017.
  66. Nicod E, Kanavos P. Scientific and social value judgments for orphan drugs in health technology assessment. Int J Technol Assess Health Care. 2016;32(4):218-232. doi:10.1017/s0266462316000416
  67. Silva EN, Sousa TR. Economic evaluation in the context of rare diseases: is it possible? Cad Saude Publica. 2015;31(3):496-506.
  68. Bertram MY, Lauer JA, De Joncheere K, et al. Cost-effectiveness thresholds: pros and cons. Bull World Health Organ. 2016;94(12):925-930. doi:10.2471/blt.15.164418
  69. Birch S, Gafni A. The biggest bang for the buck or bigger bucks for the bang: the fallacy of the cost-effectiveness threshold. J Health Serv Res Policy. 2006;11(1):46-51. doi:10.1258/135581906775094235
  70. Schlander M, Dintsios CM, Gandjour A. Budgetary impact and cost drivers of drugs for rare and ultrarare diseases. Value Health. 2018;21(5):525-531. doi:10.1016/j.jval.2017.10.015
  71. Schuller Y, Hollak CE, Biegstraaten M. The quality of economic evaluations of ultra-orphan drugs in Europe - a systematic review. Orphanet J Rare Dis. 2015;10:92. doi:10.1186/s13023-015-0305-y
  72. Drummond M, Knies S, Garau M. Sustainable funding and fair pricing for orphan drugs: what are the solutions? Presented at: ISPOR 20th Annual European Congress; 2017 Nov 4-8. Glasgow, Scotland.
  73. Linley WG, Hughes DA. Societal views on NICE, cancer drugs fund and value-based pricing criteria for prioritising medicines: a cross-sectional survey of 4118 adults in Great Britain. Health Econ. 2013;22(8):948-964. doi:10.1002/hec.2872
  74. Wiss J, Levin LA, Andersson D, Tinghog G. Prioritizing rare diseases: psychological effects influencing medical decision making. Med Decis Making. 2017;37(5):567-576. doi:10.1177/0272989x17691744
  75. Desser AS, Gyrd-Hansen D, Olsen JA, Grepperud S, Kristiansen IS. Societal views on orphan drugs: cross sectional survey of Norwegians aged 40 to 67. BMJ. 2010;341:c4715. doi:10.1136/bmj.c4715
  76. Dragojlovic N, Rizzardo S, Bansback N, Mitton C, Marra CA, Lynd LD. Challenges in measuring the societal value of orphan drugs: insights from a Canadian stated preference survey. Patient. 2015;8(1):93-101. doi:10.1007/s40271-014-0109-5
  77. Mentzakis E, Stefanowska P, Hurley J. A discrete choice experiment investigating preferences for funding drugs used to treat orphan diseases: an exploratory study. Health Econ Policy Law. 2011;6(3):405-433. doi:10.1017/s1744133110000344
  78. Desser AS. Prioritizing treatment of rare diseases: a survey of preferences of Norwegian doctors. Soc Sci Med. 2013;94:56-62. doi:10.1016/j.socscimed.2013.06.019
  79. EURORDIS. Rare diseases: Understanding this public health priority. EURORDIS website.  Updated 2005. Accessed October 21, 2013.
  80. Abelson J, Wagner F, DeJean D, et al. Public and patient involvement in health technology assessment: a framework for action. Int J Technol Assess Health Care. 2016;32(4):256-264. doi:10.1017/s0266462316000362
  81. Gagnon MP, Desmartis M, Lepage-Savary D, et al. Introducing patients' and the public's perspectives to health technology assessment: A systematic review of international experiences. Int J Technol Assess Health Care. 2011;27(1):31-42. doi:10.1017/s0266462310001315
  82. Rowe G, Frewer LJ. A typology of public engagement mechanisms. Sci Technol Human Values. 2005;30(2):251-290. doi:10.1177/0162243904271724
  83. Douglas CM, Wilcox E, Burgess M, Lynd LD. Why orphan drug coverage reimbursement decision-making needs patient and public involvement. Health Policy. 2015;119(5):588-596. doi:10.1016/j.healthpol.2015.01.009
  84. Facey K, Granados A, Guyatt G, et al. Generating health technology assessment evidence for rare diseases. Int J Technol Assess Health Care. 2014;30(4):416-422. doi:10.1017/s0266462314000464
  85. Menon D, Stafinski T, Dunn A, Wong-Rieger D. Developing a patient-directed policy framework for managing orphan and ultra-orphan drugs throughout their lifecycle. Patient. 2015;8(1):103-117. doi:10.1007/s40271-014-0108-6
  86. Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R. Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in Thailand. Health Res Policy Syst. 2012;10:6. doi:10.1186/1478-4505-10-6
  87. Goetghebeur MM, Wagner M, Khoury H, Rindress D, Gregoire JP, Deal C. Combining multicriteria decision analysis, ethics and health technology assessment: applying the EVIDEM decision-making framework to growth hormone for Turner syndrome patients. Cost Eff Resour Alloc. 2010;8:4. doi:10.1186/1478-7547-8-4
  88. Wagner M, Khoury H, Bennetts L, et al. Appraising the holistic value of Lenvatinib for radio-iodine refractory differentiated thyroid cancer: A multi-country study applying pragmatic MCDA. BMC Cancer. 2017;17(1):272. doi:10.1186/s12885-017-3258-9
  89. Christiaens W, Kohn L, Leonard C, Denis A, Daue F, Cleemput I, et al. Models for citizen and patient involvement in health care policy: Part I: Exploration of their feasibility and acceptability. Brussels: Belgian Health Care Knowledge Centre; 2012.
  90. Abrishami Shirazi P, Boer A, Horstman K. Value in co-creation: Subjecting innovative in-hospital technologies to multi-stakeholder appraisal. International Journal of Hospital Based Health Technology Assessment. 2017;1:12-30. doi:10.21965/IJHBHTA.2017.002
  91. Hofmann B, Cleemput I, Bond K, et al. Revealing and acknowledging value judgments in health technology assessment. Int J Technol Assess Health Care. 2014;30(6):579-586. doi:10.1017/s0266462314000671