Advancing the WHO-INTEGRATE Framework as a Tool for Evidence-Informed, Deliberative Decision-Making Processes: Exploring the Views of Developers and Users of WHO Guidelines

Document Type : Original Article

Authors

1 Institute for Medical Information Processing, Biometry, and Epidemiology – IBE, LMU Munich, Munich, Germany

2 Pettenkofer School of Public Health, Munich, Germany

3 Save the Children, Kathmandu, Nepal

4 The Water Institute, Department of Environmental Sciences and Engineering, The University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

5 National Supplementary Health Agency, Ministry of Health, Brasília, Brazil

6 Department of MaternalInfant Nursing and Public Health, College of Nursing, University of São Paulo, Ribeirão Preto, Brazil

7 Department of Disease Control and Environmental Health, Makerere University School of Public Health, Kampala, Uganda

8 Department of Global Health, Norwegian Institute of Public Health, Oslo, Norway

9 Bavarian Health and Food Safety Authority, Munich, Germany

Abstract

Background
Decision-making on matters of public health and health policy is a deeply value-laden process. The World Health Organization (WHO)-INTEGRATE framework was proposed as a new evidence-to-decision (EtD) framework to support guideline development from a complexity perspective, notably in relation to public health and health system interventions, and with a foundation in WHO norms and values. This study was conducted as part of the development of the framework to assess its comprehensiveness and usefulness for public health and health policy decision-making.
 
Methods
We conducted a qualitative study comprising nine key informant interviews (KIIs) with experts involved in WHO guideline development and four focus group discussions (FGDs) with a total of forty health decision-makers from Brazil, Germany, Nepal and Uganda. Transcripts were analyzed using MAXQDA12 and qualitative content analysis.
 
Results
Most key informants and participants in the FGDs appreciated the framework for its relevance to real-world decision-making on four widely differing health topics. They praised its broad perspective and comprehensiveness with respect to new or expanded criteria, notably regarding societal implications, equity considerations, and acceptability. Some guideline developers questioned the value of the framework beyond current practice and were concerned with the complexity of applying such a broad range of criteria in guideline development processes. Participants made concrete suggestions for improving the wording and definitions of criteria as well as their grouping, for covering missing aspects, and for addressing overlap between criteria.
 
Conclusion
The framework was well-received by health decision-makers as well as the developers of WHO guidelines and appears to capture all relevant considerations discussed in four distinct real-world decision processes that took place on four different continents. Guidance is needed on how to apply the framework in guideline processes that are both transparent and participatory. A set of suggestions for improvement provides a valuable starting point for advancing the framework towards version 2.0.

Keywords

References

  1. Sanderson I. Intelligent policy making for a complex world: pragmatism, evidence and learning. Polit Stud. 2009;57(4):699-719. doi:10.1111/j.1467-9248.2009.00791.x
  2. Petticrew M, Knai C, Thomas J, et al. Implications of a complexity perspective for systematic reviews and guideline development in health decision making. BMJ Glob Health. 2019;4(Suppl 1):e000899. doi:10.1136/bmjgh-2018-000899
  3. Rutter H, Savona N, Glonti K, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390(10112):2602-2604. doi:10.1016/s0140-6736(17)31267-9
  4. Lewin S, Hendry M, Chandler J, et al. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Med Res Methodol. 2017;17(1):76. doi:10.1186/s12874-017-0349-x
  5. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43(3-4):267-276. doi:10.1007/s10464-009-9229-9
  6. Baltussen R, Jansen MP, Mikkelsen E, et al. Priority setting for universal health coverage: we need evidence-informed deliberative processes, not just more evidence on cost-effectiveness. Int J Health Policy Manag. 2016;5(11):615-618. doi:10.15171/ijhpm.2016.83
  7. Baltussen R, Mikkelsen E, Tromp N, et al. Balancing efficiency, equity and feasibility of HIV treatment in South Africa - development of programmatic guidance. Cost Eff Resour Alloc. 2013;11(1):26. doi:10.1186/1478-7547-11-26
  8. Kapiriri L, Lee NM, Wallace LJ, Kwesiga B. Beyond cost-effectiveness, morbidity and mortality: a comprehensive evaluation of priority setting for HIV programming in Uganda. BMC Public Health. 2019;19(1):359. doi:10.1186/s12889-019-6690-8
  9. Guindo LA, Wagner M, Baltussen R, et al. From efficacy to equity: literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc. 2012;10(1):9. doi:10.1186/1478-7547-10-9
  10. Youngkong S, Kapiriri L, Baltussen R. Setting priorities for health interventions in developing countries: a review of empirical studies. Trop Med Int Health. 2009;14(8):930-939. doi:10.1111/j.1365-3156.2009.02311.x
  11. United Nations. Transforming our World: The 2030 Agenda for Sustainable Development. New York: United Nations; 2015.
  12. Holm S. The second phase of priority setting. Goodbye to the simple solutions: the second phase of priority setting in health care. BMJ. 1998;317(7164):1000-1002.
  13. Kapiriri L, Martin DK. A strategy to improve priority setting in developing countries. Health Care Anal. 2007;15(3):159-167. doi:10.1007/s10728-006-0037-1
  14. Baltussen R, Jansen MPM, Bijlmakers L, et al. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256-260. doi:10.1016/j.jval.2016.11.019
  15. Norris SL, Rehfuess EA, Smith H, et al. Complex health interventions in complex systems: improving the process and methods for evidence-informed health decisions. BMJ Glob Health. 2019;4(Suppl 1):e000963. doi:10.1136/bmjgh-2018-000963
  16. Rehfuess EA, Stratil JM, Scheel IB, Portela A, Norris SL, Baltussen R. The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective. BMJ Glob Health. 2019;4(Suppl 1):e000844. doi:10.1136/bmjgh-2018-000844
  17. Daniels N, Sabin J. The ethics of accountability in managed care reform. Health Aff (Millwood). 1998;17(5):50-64. doi:10.1377/hlthaff.17.5.50
  18. Daniels N. Accountability for reasonableness. BMJ. 2000;321(7272):1300-1301. doi:10.1136/bmj.321.7272.1300
  19. Daniels N, Sabin JE. Accountability for reasonableness: an update BMJ. 2008;337:a1850. doi:10.1136/bmj.a1850
  20. Daniels N, Sabin J. Just Health: Meeting Health Needs Fairly. 1st ed. Cambridge: Cambridge University Press; 2008.
  21. Daniels N, Sabin J. The ethics of accountability in managed care reform. Health Aff (Millwood). 1998;17(5):50-64. doi:10.1377/hlthaff.17.5.50
  22. Friedman A. Beyond accountability for reasonableness. Bioethics. 2008;22(2):101-112. doi:10.1111/j.1467-8519.2007.00605.x
  23. Singer PA, Martin DK, Giacomini M, Purdy L. Priority setting for new technologies in medicine: qualitative case study. BMJ. 2000;321(7272):1316-1318. doi:10.1136/bmj.321.7272.1316
  24. Tyler TR. Psychological perspectives on legitimacy and legitimation. Annu Rev Psychol. 2006;57:375-400. doi:10.1146/annurev.psych.57.102904.190038
  25. Greenberg J, Folger R. Procedural justice, participation, and the fair process effect in groups and organizations. In: Paulus PB, eds. Basic Group Processes. New York, NY: Springer; 1983:235-256. doi:10.1007/978-1-4612-5578-9_10
  26. Saltelli A, Giampietro M. What is wrong with evidence based policy, and how can it be improved? Futures. 2017;91:62-71. doi:10.1016/j.futures.2016.11.012
  27. World Health Organization (WHO). WHO Handbook for Guideline Development. 2nd ed. Geneva, Switzerland: WHO; 2014.
  28. Ham C. Priority setting in health care: learning from international experience. Health Policy. 1997;42(1):49-66. doi:10.1016/s0168-8510(97)00054-7
  29. Niessen LW, Bridges J, Lau BD, et al. Assessing the Impact of Economic Evidence on Policymakers in Health Care-A Systematic Review. Rockville, MD: Agency for Healthcare Research and Quality (US); 2012.
  30. Alonso-Coello P, Schünemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016. doi:10.1136/bmj.i2016
  31. Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc. 2006;4:14. doi:10.1186/1478-7547-4-14
  32. Nutt PC. Surprising but true: half the decisions in organizations fail. Acad Manag Perspect. 1999;13(4):75-90. doi:10.5465/ame.1999.2570556
  33. Robinson R. Limits to rationality: economics, economists and priority setting. Health Policy. 1999;49(1-2):13-26. doi:10.1016/s0168-8510(99)00040-8
  34. Gigerenzer G. Decision making: nonrational theories. In: Smelser NJ, Baltes PB, eds. International Encyclopedia of the Social & Behavioral Sciences. Oxford: Elsevier Science; 2001:3304-3309.
  35. Moberg J, Oxman AD, Rosenbaum S, et al. The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Syst. 2018;16(1):45. doi:10.1186/s12961-018-0320-2
  36. Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet. 2017;390(10092):415-423. doi:10.1016/s0140-6736(16)31592-6
  37. Gopinathan U, Hoffman SJ. Institutionalising an evidence-informed approach to guideline development: progress and challenges at the World Health Organization. BMJ Glob Health. 2018;3(5):e000716. doi:10.1136/bmjgh-2018-000716
  38. Stratil JM, Baltussen R, Scheel I, Nacken A, Rehfuess EA. Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making. Cost Eff Resour Alloc. 2020;18:8. doi:10.1186/s12962-020-0203-6
  39. World Health Organization (WHO). Consolidated Guideline on Sexual and Reproductive Health and Rights of Women Living with HIV. Geneva, Switzerland: WHO; 2017.
  40. World Health Organization (WHO). Communicating Risk in Public Health Emergencies: A WHO Guideline for Emergency Risk Communication (ERC) Policy and Practice. Geneva, Switzerland: WHO; 2018.
  41. World Health Organization (WHO). WHO Recommendations on Antenatal Care for a Positive Pregnancy Experience. Geneva, Switzerland: WHO; 2014.
  42. Mayring P. Qualitative Content Analysis: Theoretical Foundation, Basic Procedures and Software Solution. Klagenfurt, Austria: Social Science Open Access Repository; 2014.
  43. Cope MB, Allison DB. White hat bias: examples of its presence in obesity research and a call for renewed commitment to faithfulness in research reporting. Int J Obes (Lond). 2010;34(1):84-88. doi:10.1038/ijo.2009.239
  44. Cromwell I, Peacock SJ, Mitton C. 'Real-world' health care priority setting using explicit decision criteria: a systematic review of the literature. BMC Health Serv Res. 2015;15:164. doi:10.1186/s12913-015-0814-3
  45. Noorani HZ, Husereau DR, Boudreau R, Skidmore B. Priority setting for health technology assessments: a systematic review of current practical approaches. Int J Technol Assess Health Care. 2007;23(3):310-315. doi:10.1017/s026646230707050x
  46. Asante AD, Zwi AB. Factors influencing resource allocation decisions and equity in the health system of Ghana. Public Health. 2009;123(5):371-377. doi:10.1016/j.puhe.2009.02.006
  47. Lasry A, Carter MW, Zaric GS. Allocating funds for HIV/AIDS: a descriptive study of KwaDukuza, South Africa. Health Policy Plan. 2011;26(1):33-42. doi:10.1093/heapol/czq022
  48. Baltussen R, Stolk E, Chisholm D, Aikins M. Towards a multi-criteria approach for priority setting: an application to Ghana. Health Econ. 2006;15(7):689-696. doi:10.1002/hec.1092
  49. Baltussen R, ten Asbroek AH, Koolman X, Shrestha N, Bhattarai P, Niessen LW. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Health Policy Plan. 2007;22(3):178-185. doi:10.1093/heapol/czm010
  50. World Health Organization (WHO). Everybody's Business -- Strengthening Health Systems to Improve Health Outcomes: WHO's Framework for Action. Geneva: WHO; 2007.
  51. Ryan MA. Adherence to clinical practice guidelines. Otolaryngol Head Neck Surg. 2017;157(4):548-550. doi:10.1177/0194599817718822
  52. Fischer F, Lange K, Klose K, Greiner W, Kraemer A. Barriers and strategies in guideline implementation-a scoping review. Healthcare (Basel). 2016;4(3). doi:10.3390/healthcare4030036
  53. McDonald S, Elliott JH, Green S, Turner T. Towards a new model for producing evidence-based guidelines: a qualitative study of current approaches and opportunities for innovation among Australian guideline developers. F1000Res. 2019;8:956. doi:10.12688/f1000research.19661.1
  54. Turner TJ. Developing evidence-based clinical practice guidelines in hospitals in Australia, Indonesia, Malaysia, the Philippines and Thailand: values, requirements and barriers. BMC Health Serv Res. 2009;9:235. doi:10.1186/1472-6963-9-235
  55. Florez ID, Morgan RL, Falavigna M, et al. Development of rapid guidelines: 2. A qualitative study with WHO guideline developers. Health Res Policy Syst. 2018;16(1):62. doi:10.1186/s12961-018-0329-6
  56. Setty K, Jiménez A, Willetts J, Leifels M, Bartram J. Global water, sanitation and hygiene research priorities and learning challenges under Sustainable Development Goal 6. Dev Policy Rev. 2020;38(1):64-84. doi:10.1111/dpr.12475
  57. Tromp N, Prawiranegara R, Siregar A, et al. Translating international HIV treatment guidelines into local priorities in Indonesia. Trop Med Int Health. 2018;23(3):279-294. doi:10.1111/tmi.13031
  58. Heintz E, Lintamo L, Hultcrantz M, et al. Framework for systematic identification of ethical aspects of healthcare technologies: the SBU approach. Int J Technol Assess Health Care. 2015;31(3):124-130. doi:10.1017/s0266462315000264
  59. Norheim OF, Baltussen R, Johri M, et al. Guidance on priority setting in health care (GPS-Health): the inclusion of equity criteria not captured by cost-effectiveness analysis. Cost Eff Resour Alloc. 2014;12:18. doi:10.1186/1478-7547-12-18
  60. Johnson AP, Sikich NJ, Evans G, et al. Health technology assessment: a comprehensive framework for evidence-based recommendations in Ontario. Int J Technol Assess Health Care. 2009;25(2):141-150. doi:10.1017/s0266462309090199
  61. Treweek S, Oxman AD, Alderson P, et al. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6. doi:10.1186/1748-5908-8-6
  62. Alonso-Coello P, Schünemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016. doi:10.1136/bmj.i2016
  63. Vogel JP, Oxman AD, Glenton C, et al. Policymakers' and other stakeholders' perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: an international survey. Health Res Policy Syst. 2013;11:19. doi:10.1186/1478-4505-11-19
  64. Hennink MM, Kaiser BN, Marconi VC. Code saturation versus meaning saturation: how many interviews are enough? Qual Health Res. 2017;27(4):591-608. doi:10.1177/1049732316665344
  65. Stratil JM, Baltussen R, Scheel I, Nacken A, Rehfuess EA. Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making. Cost Eff Resour Alloc. 2020;18:8. doi:10.1186/s12962-020-0203-6

Files

History

  • Receive Date: 30 March 2020
  • Revise Date: 15 August 2020
  • Accept Date: 29 September 2020
  • First Publish Date: 27 October 2020

Share

How to cite

Statistics