Characterizing the Validity and Real-World Utility of Health Technology Assessments in Healthcare: Future Directions; Comment on “Problems and Promises of Health Technologies: The Role of Early Health Economic Modelling”

Document Type: Commentary


1 Schaeffer Center for Health Policy and Economics, Department of Pharmaceutical and Health Economics, School of Pharmacy, University of Southern California, Los Angeles, CA, USA

2 USC Clinical Economics Research and Education Program (CEREP), Los Angeles, CA, USA


With their article, Grutters et al raise an important question: What do successful health technology assessments (HTAs) look like, and what is their real-world utility in decision-making? While many HTAs are published in peer-reviewed journals, many are considered proprietary and their attributes remain confidential, limiting researchers’ ability to answer these questions. Models for economic evaluations like cost-effectiveness analyses (CEAs) synthesize a wide range of evidence, are often statistically and mathematically sophisticated, and require untestable assumptions. As such, there is nearly universal agreement among researchers that enhancing transparency is an important issue in health economic modeling. However, the definition of transparency and guidelines for its implementation vary. Model registration combined with a linked database of model-based economic evaluations has been proposed as a solution, whereby registered models and their accompanying technical and nontechnical documentation are sourced into a single publicly-available repository, ideally in a standardized format to ensure consistent and complete representation of features, code, data sources, results, validation exercises, and policy recommendations. When such a repository is ultimately created, modelers will not have to reinvent the wheel for every new drug launched or new treatment pathway. These more open and transparent approaches will have substantial implications for model accuracy, reliability, and validity, improving trust and acceptance by healthcare decision-makers.


Main Subjects

  1. Grutters JPC, Govers T, Nijboer J, Tummers M, van der Wilt GJ, Rovers MM. Problems and promises of health technologies: the role of early health economic modeling. Int J Health Policy Manag. 2019;8(10):575-582. doi:10.15171/ijhpm.2019.36
  2. Mandelblatt JS, Fryback DG, Weinstein MC, Russell LB, Gold MR, et al. Assessing the effectiveness of health interventions for cost-effectiveness analysis. In: Gold MR, Siegel JE, Russell LB, Weinstein MC, et al, eds. Cost-effectiveness analysis in health and medicine. New York (NY): Oxford University Press; 1996:135-164.
  3. McCabe C, Dixon S. Testing the validity of cost-effectiveness models. Pharmacoeconomics. 2000;17(5):501-513. doi:10.2165/00019053-200017050-00007
  4. Ofman JJ, Sullivan SD, Neumann PJ, et al. Examining the value and quality of health economic analyses: implications of utilizing the QHES. J Manag Care Pharm. 2003;9(1):53-61. doi:10.18553/jmcp.2003.9.1.53
  5. Goodacre S, McCabe C. Being economical with the truth: how to make your idea appear cost effective. Emerg Med J. 2002;19(4):301-304. doi:10.1136/emj.19.4.301
  6. Hay JW. Now Is the Time for Transparency in Value-Based Healthcare Decision Modeling. Value Health. 2019;22(5):564-569. doi:10.1016/j.jval.2019.01.009
  7. Neumann PJ, Kim DD, Trikalinos TA, et al. Future directions for cost-effectiveness analyses in health and medicine. Med Decis Making. 2018;38(7):767-777. doi:10.1177/0272989x18798833
  8. Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Med Decis Making. 2012;32(5):733-743. doi:10.1177/0272989x12454579
  9. Husereau D, Drummond M, Petrou S, et al. Consolidated health economic evaluation reporting standards (CHEERS)--explanation and elaboration: a report of the ISPOR health economic evaluation publication guidelines good reporting practices task force. Value Health. 2013;16(2):231-250. doi:10.1016/j.jval.2013.02.002
  10. Cohen AB. Point-counterpoint: cost-effectiveness analysis in medical care and the issue of economic model transparency. Med Care. 2017;55(11):907-908. doi:10.1097/mlr.0000000000000812
  11. Padula WV, McQueen RB, Pronovost PJ. Can economic model transparency improve provider interpretation of cost-effectiveness analysis? evaluating tradeoffs presented by the second panel on cost-effectiveness in health and medicine. Med Care. 2017;55(11):909-911. doi:10.1097/mlr.0000000000000810
  12. Cohen JT, Wong JB. Can economic model transparency improve provider interpretation of cost-effectiveness analysis? a response. Med Care. 2017;55(11):912-914. doi:10.1097/mlr.0000000000000811
  13. Morin A, Urban J, Adams PD, et al. Research priorities. Shining light into black boxes. Science. 2012;336(6078):159-160. doi:10.1126/science.1218263
  14. Shepard DS. Cost-effectiveness in Health and Medicine. By M.R. Gold, J.E Siegel, L.B. Russell, and M.C. Weinstein (eds). New York: Oxford University Press, 1996. J Ment Health Policy Econ. 1999;2(2):91-92. doi:10.1002/(sici)1099-176x(199906)2:2<91::aid-mhp46>;2-i
  15. Kane JP, Malloy MJ, Ports TA, Phillips NR, Diehl JC, Havel RJ. Regression of coronary atherosclerosis during treatment of familial hypercholesterolemia with combined drug regimens. JAMA. 1990;264(23):3007-3012.
  16. Caruzzo C, Liboni W, Bonzano A, et al. Effect of lipid-lowering treatment on progression of atherosclerotic lesions--a duplex ultrasonographic investigation. Angiology. 1995;46(4):269-280. doi:10.1177/000331979504600401
  17. Guyton JR, Brown BG, Fazio S, Polis A, Tomassini JE, Tershakovec AM. Lipid-altering efficacy and safety of ezetimibe/simvastatin coadministered with extended-release niacin in patients with type IIa or type IIb hyperlipidemia. J Am Coll Cardiol. 2008;51(16):1564-1572. doi:10.1016/j.jacc.2008.03.003
  18. Sang ZC, Wang F, Zhou Q, et al. Combined use of extended-release niacin and atorvastatin: safety and effects on lipid modification. Chin Med J (Engl). 2009;122(14):1615-1620.
  19. Boden WE, Probstfield JL, Anderson T, et al. Niacin in patients with low HDL cholesterol levels receiving intensive statin therapy. N Engl J Med. 2011;365(24):2255-2267. doi:10.1056/NEJMoa1107579
  20. Landray MJ, Haynes R, Hopewell JC, et al. Effects of extended-release niacin with laropiprant in high-risk patients. N Engl J Med. 2014;371(3):203-212. doi:10.1056/NEJMoa1300955
  21. D'Andrea E, Hey SP, Ramirez CL, Kesselheim AS. Assessment of the Role of Niacin in Managing Cardiovascular Disease Outcomes: A Systematic Review and Meta-analysis. JAMA Netw Open. 2019;2(4):e192224. doi:10.1001/jamanetworkopen.2019.2224
  22. Oortwijn W, Sampietro-Colom L, Trowman R. How to Deal with the Inevitable: Generating Real-World Data and Using Real-World Evidence for HTA Purposes - From Theory to Action. Int J Technol Assess Health Care. 2019;35(4):346-350. doi:10.1017/s0266462319000400
  23. Makady A, van Veelen A, Jonsson P, et al. Using real-world data in health technology assessment (HTA) practice: a comparative study of five HTA agencies. Pharmacoeconomics. 2018;36(3):359-368. doi:10.1007/s40273-017-0596-z
  24. Hartford J, Lewis G, Leyton-Brown K, Taddy M. Deep IV: A Flexible Approach for Counterfactual Prediction. Proceedings of the 34th International Conference on Machine Learning; 2017.
  25. VanderWeele TJ, Ding P. Sensitivity analysis in observational research: introducing the E-value. Ann Intern Med. 2017;167(4):268-274. doi:10.7326/m16-2607
  26. McManus E, Turner D, Sach T. Can you repeat that? exploring the definition of a successful model replication in health economics. Pharmacoeconomics. 2019;37(11):1371-1381. doi:10.1007/s40273-019-00836-y
  27. Palmer AJ, Si L, Tew M, et al. Computer modeling of diabetes and its transparency: a report on the eighth mount hood challenge. Value Health. 2018;21(6):724-731. doi:10.1016/j.jval.2018.02.002
  28. The Mt Hood Diabetes Challenge.  Accessed November 26, 2019.
  29. Shao H, Fonseca V, Stoecker C, Liu S, Shi L. Novel risk engine for diabetes progression and mortality in USA: building, relating, assessing, and validating outcomes (BRAVO). Pharmacoeconomics. 2018;36(9):1125-1134. doi:10.1007/s40273-018-0662-1
  30. BRAVO Model.  Accessed November 26, 2019.
  31. Shao H, Yang S, Stoecker C, Fonseca V, Hong D, Shi L. Addressing Regional Differences in Diabetes Progression: Global Calibration for Diabetes Simulation Model. Value Health. 2019. doi:10.1016/j.jval.2019.08.007
  32. Kent S, Becker F, Feenstra T, et al. The challenge of transparency and validation in health economic decision modelling: a view from Mount Hood. Pharmacoeconomics. 2019;37(11):1305-1312. doi:10.1007/s40273-019-00825-1
  33. Diabetes simulation modeling database.  Accessed November 26, 2019.