Fostering Responsible Innovation in Health: An EvidenceInformed Assessment Tool for Innovation Stakeholders

Document Type : Original Article

Authors

1 Public Health Research Institute, University of Montreal, Montreal, QC, Canada

2 School of Public Health, University of Montreal, Montreal, QC, Canada

Abstract

Background
Responsible innovation in health (RIH) emphasizes the importance of developing technologies that are responsive to system-level challenges and support equitable and sustainable healthcare. To help decision-makers identify whether an innovation fulfills RIH requirements, we developed and validated an evidence-informed assessment tool comprised of 4 inclusion and exclusion criteria, 9 assessment attributes and a scoring system.

 
Methods
We conducted an inter-rater reliability assessment to establish the extent to which 2 raters agree when applying the RIH Tool to a diversified sample of health innovations (n = 25). Following the Tool’s 3-step process, sources of information were collected and cross-checked to ensure their clarity and relevance. Ratings were reported independently in a spreadsheet to generate the study’s database. To measure inter-rater reliability, we used: a non-adjusted index (percent agreement), a chance-adjusted index (Gwet’s AC) and the Pearson’s correlation coefficient. Results of the Tool’s application to the whole sample of innovations are summarized through descriptive statistics.

 
Results
Our findings show complete agreement for the screening criteria, “almost perfect” agreement for 7 assessment attributes, “substantial” agreement for 2 attributes and “almost perfect” agreement for the RIH overall score. A large portion of the sample obtained high scores for 6 attributes (health relevance, health inequalities, responsiveness, level and intensity of care and frugality) and low scores for 3 attributes (ethical, legal, and social issues [ELSIs], inclusiveness and eco-responsibility). At the rating step, 88% of the innovations had a sufficient number of attributes documented (≥ 7/9), but the assessment was based on sources of moderate to high quality (mean score ≥ 2 points) for 36% of the sample. While “Almost all RIH features” were present for 24% of the innovations (RIH mean score between 4.1-5.0 points), “Many RIH features” were present for 52% of the sample (3.1-4.0 points) and “Few RIH features” were present for 24% of the innovations (2.1-3.0 points).

 
Conclusion
By confirming key aspects of the RIH Tool’s reliability and applicability, our study brings its development to completion. It can be jointly put into action by innovation stakeholders who want to foster innovations with greater social, economic and environmental value.

Highlights

Supplementary File 1 (Download)

Keywords

References

  1. Fineberg HV. Shattuck Lecture. A successful and sustainable health system--how to get there from here. N Engl J Med. 2012;366(11):1020-1027. doi:10.1056/NEJMsa1114777
  2. Garber S, Gates SM, Keeler EB, et al. Redirecting innovation in U.S. health care: options to decrease spending and increase value. Rand Health Q. 2014;4(1):3.
  3. Lehoux P, Roncarolo F, Silva HP, Boivin A, Denis JL, Hebert R. What health system challenges should responsible innovation in health address? insights from an international scoping review. Int J Health Policy Manag. 2019;8(2):63-75. doi:10.15171/ijhpm.2018.110
  4. Macdonnell M, Darzi A. A key to slower health spending growth worldwide will be unlocking innovation to reduce the labor-intensity of care. Health Aff (Millwood). 2013;32(4):653-660. doi:10.1377/hlthaff.2012.1330
  5. Tony M, Wagner M, Khoury H, et al. Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Serv Res. 2011;11:329. doi:10.1186/1472-6963-11-329
  6. Marsh KD, Sculpher M, Caro JJ, Tervonen T. The use of MCDA in HTA: great potential, but more effort needed. Value Health. 2018;21(4):394-397. doi:10.1016/j.jval.2017.10.001
  7. Marsh K, Lanitis T, Neasham D, Orfanos P, Caro J. Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. Pharmacoeconomics. 2014;32(4):345-365. doi:10.1007/s40273-014-0135-0
  8. Marsh K, Goetghebeur M, Thokala P, Baltussen R. Multi-Criteria Decision Analysis to Support Healthcare Decisions. London, UK: Springer; 2017.
  9. Adunlin G, Diaby V, Xiao H. Application of multicriteria decision analysis in health care: a systematic review and bibliometric analysis. Health Expect. 2015;18(6):1894-1905. doi:10.1111/hex.12287
  10. Diaby V, Campbell K, Goeree R. Multi-criteria decision analysis (MCDA) in health care: a bibliometric analysis. Oper Res Health Care. 2013;2(1-2):20-24. doi:10.1016/j.orhc.2013.03.001
  11. Goetghebeur MM, Cellier MS. Can reflective multicriteria be the new paradigm for healthcare decision-making? the EVIDEM journey. Cost Eff Resour Alloc. 2018;16(Suppl 1):54. doi:10.1186/s12962-018-0116-9
  12. Wahlster P, Goetghebeur M, Kriza C, Niederlander C, Kolominsky-Rabas P. Balancing costs and benefits at different stages of medical innovation: a systematic review of Multi-criteria decision analysis (MCDA). BMC Health Serv Res. 2015;15:262. doi:10.1186/s12913-015-0930-0
  13. von Schomberg R. A Vision of responsible research and innovation. In: Owen R, Bessant J, Heintz M, eds. Responsible Innovation: Managing the Responsible Emergence of Science and Innovation in Society. London: Wiley; 2013:51-74.
  14. Stilgoe J, Owen R, Macnaghten P. Developing a framework for responsible innovation. Res Policy. 2013;42(9):1568-1580. doi:10.1016/j.respol.2013.05.008
  15. Lehoux P, Pacifico Silva H, Pozelli Sabio R, Roncarolo F. The unexplored contribution of responsible innovation in health to Sustainable Development Goals. Sustainability. 2018;10(11):4015. doi:10.3390/su10114015
  16. Pacifico Silva H, Lehoux P, Miller FA, Denis JL. Introducing responsible innovation in health: a policy-oriented framework. Health Res Policy Syst. 2018;16(1):90. doi:10.1186/s12961-018-0362-5
  17. Stahl BC. Who is responsible for responsible innovation? lessons from an investigation into responsible innovation in health: Comment on "What health system challenges should responsible innovation in health address? Insights from an international scoping review." Int J Health Policy Manag. 2019;8(7):447-449. doi:10.15171/ijhpm.2019.32
  18. Pacifico Silva H, Lehoux P, Hagemeister N. Developing a tool to assess responsibility in health innovation: Results from an international Delphi study. Health Policy Technol. 2018;7(4):388-396. doi:10.1016/j.hlpt.2018.10.007
  19. Bolarinwa OA. Principles and methods of validity and reliability testing of questionnaires used in social and health science researches. Niger Postgrad Med J. 2015;22(4):195-201. doi:10.4103/1117-1936.173959
  20. Gwet KL. Handbook of Inter-Rater Reliability. 4th ed. Advanced Analytics, LLC; 2014.
  21. Zhao X, Liu JS, Deng K. Assumption behind intercoder reliability indices. In: Salmon CT, ed. Communication Yearbook 36. New York: Routledge; 2013:419-480.
  22. Quarfoot D, Levine RA. How robust are multirater interrater reliability indices to changes in frequency distribution? Am Stat. 2016;70(4):373-384. doi:10.1080/00031305.2016.1141708
  23. Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen's Kappa and Gwet's AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13:61. doi:10.1186/1471-2288-13-61
  24. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543-549. doi:10.1016/0895-4356(90)90158-l
  25. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174.
  26. Hinkle DE, Wiersma W, Jurs SG. Applied Statistics for the Behavioral Sciences. Houghton Mifflin; 2003.
  27. Polisena J, De Angelis G, Kaunelis D, Gutierrez-Ibarluzea I. Environmental impact assessment of a health technology: a scoping review. Int J Technol Assess Health Care. 2018;34(3):317-326. doi:10.1017/s0266462318000351
  28. Campion N, Thiel CL, Woods NC, Swanzy L, Landis AE, Bilec MM. Sustainable healthcare and environmental life-cycle impacts of disposable supplies: a focus on disposable custom packs. J Clean Prod. 2015;94:46-55. doi:10.1016/j.jclepro.2015.01.076
  29. Garau M, Devlin NJ. Using MCDA as a decision aid in health technology appraisal for coverage decisions: opportunities, challenges and unresolved questions. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, eds. Multi-Criteria Decision Analysis to Support Healthcare Decisions. Cham: Springer; 2017:277-298. doi:10.1007/978-3-319-47540-0_14
  30. Culyer AJ. Deliberative Processes in Decisions about Health Care Technologies: Combining Different Types of Evidence, Values, Algorithms and People. OHE Briefing, No. 48, June 2009. https://ssrn.com/abstract=2640171.
  31. Weyrauch T, Herstatt C. What is frugal innovation? three defining criteria. J Frugal Innov. 2016;2(1):1. doi:10.1186/s40669-016-0005-y
  32. Davis M, Laas K. "Broader impacts" or "responsible research and innovation"? a comparison of two criteria for funding research in science and engineering. Sci Eng Ethics. 2014;20(4):963-983. doi:10.1007/s11948-013-9480-1
  33. Robinson JC. Biomedical innovation in the era of health care spending constraints. Health Aff (Millwood). 2015;34(2):203-209. doi:10.1377/hlthaff.2014.0975
  34. Miller FA, Lehoux P, Peacock S, et al. How procurement judges the value of medical technologies: a review of healthcare tenders. Int J Technol Assess Health Care. 2019;35(1):50-55. doi:10.1017/s0266462318003756
  35. Rivard L, Lehoux P. When desirability and feasibility go hand in hand: innovators’ perspectives on what is and is not responsible innovation in health. J Responsible Innov. 2020;7(1):76-95. doi:10.1080/23299460.2019.1622952
International Journal of Health Policy and Management
Volume 10, Issue 4
April 2021
Pages 181-191

Files

History

  • Receive Date: 02 August 2019
  • Revise Date: 22 December 2019
  • Accept Date: 26 February 2020
  • First Publish Date: 01 April 2021

Share

How to cite

Statistics