Lost in Translation: Piloting a Novel Framework to Assess the Challenges in Translating Scientific Uncertainty From Empirical Findings to WHO Policy Statements

Document Type: Original Article


1 Institute for Health and Social Policy, McGill University, Montreal, QC, Canada

2 Department of Family Medicine and Public Health & Scripps Institution of Oceanography, University of California, San Diego, CA, USA

3 Chaire approches communautaires et inégalités de santé, Institut de recherche en santé publique, École de santé publique, Université de Montréal, Montreal, QC, Canada


Calls for evidence-informed public health policy, with implicit promises of greater program effectiveness, have intensified recently. The methods to produce such policies are not self-evident, requiring a conciliation of values and norms between policy-makers and evidence producers. In particular, the translation of uncertainty from empirical research findings, particularly issues of statistical variability and generalizability, is a persistent challenge because of the incremental nature of research and the iterative cycle of advancing knowledge and implementation. This paper aims to assess how the concept of uncertainty is considered and acknowledged in World Health Organization (WHO) policy recommendations and guidelines.
We selected four WHO policy statements published between 2008-2013 regarding maternal and child nutrient supplementation, infant feeding, heat action plans, and malaria control to represent topics with a spectrum of available
evidence bases. Each of these four statements was analyzed using a novel framework to assess the treatment of statistical variability and generalizability.
WHO currently provides substantial guidance on addressing statistical variability through GRADE (Grading of Recommendations Assessment, Development, and Evaluation) ratings for precision and consistency in their guideline documents. Accordingly, our analysis showed that policy-informing questions were addressed by systematic reviews and representations of statistical variability (eg, with numeric confidence intervals). In contrast, the presentation of contextual or “background” evidence regarding etiology or disease burden showed little consideration for this variability. Moreover, generalizability or “indirectness” was uniformly neglected, with little explicit consideration of study settings or subgroups.
In this paper, we found that non-uniform treatment of statistical variability and generalizability factors that may contribute to uncertainty regarding recommendations were neglected, including the state of evidence informing background questions (prevalence, mechanisms, or burden or distributions of health problems) and little assessment of generalizability, alternate interventions, and additional outcomes not captured by systematic review. These other factors often form a basis for providing policy recommendations, particularly in the absence of a strong evidence base for intervention effects. Consequently, they should also be subject to stringent and systematic evaluation criteria. We suggest that more effort is needed to systematically acknowledge (1) when evidence is missing, conflicting, or equivocal, (2) what normative considerations were also employed, and (3) how additional evidence may be accrued.


Main Subjects

  1. Rychetnik L, Hawe P, Waters E, Barratt A, Frommer M. A glossary for evidence based public health. J Epidemiol Community Health. 2004;58(7):538-545.
  2. Brownson RC, Chriqui JF, Stamatakis KA. Understanding evidence-based public health policy. Am J Public Health. 2009;99(9):1576-1583.
  3. Oliver K, Lorenc T, Innvær S. New directions in evidence-based policy research: a critical analysis of the literature. Health Research Policy and Systems. 2014;12:34. doi:10.1186/1478-4505-12-34
  4. Bayer R, Johns DM, Galea S. Salt and public health: contested science and the challenge of evidence-based decision making. Health Aff (Millwood). 2012;31(12):2738-2746. doi:10.1377/hlthaff.2012.0554
  5. Knaggård Å. What do policy-makers do with scientific uncertainty? The incremental character of Swedish climate change policy-making. Policy Stud. 2014;35(1):22-39.
  6. Dobrow MJ, Goel V, Lemieux-Charles L, Black NA. The impact of context on evidence utilization: a framework for expert groups developing health policy recommendations. Soc Sci  Med. 2006;63(7):1811-1824.
  7. Lee K, Collinson S, Walt G, Gilson L. Who should be doing what in the international health: a confusion of mandates in the United Nations? BMJ. 1996;312(7026):302.
  8. Gostin LO, Sridhar D, Hougendobler D. The normative authority of the World Health Organization. Public Health. 2015;129(7):854-863. doi:10.1016/j.puhe.2015.05.002
  9. Oxman AD, Schünemann HJ, Fretheim A. Improving the use of research evidence in guideline development: 14. Reporting guidelines. Health Res Policy Syst. 2006;4:26. doi:10.1186/1478-4505-4-26
  10. Oxman AD, Lavis JN, Fretheim A. Use of evidence in WHO recommendations. Lancet. 2007;369(9576):1883-1889.
  11. Innvær S, Vist G, Trommald M, Oxman A. Health policy-makers' perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7(4):239-244. doi:10.1258/135581902320432778
  12. Orton L, Lloyd-Williams F, Taylor-Robinson D, O'Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PloS One. 2011;6(7):e21704. doi:10.1371/journal.pone.0021704
  13. Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2. doi:10.1186/1472-6963-14-2
  14. Frenk J, Moon S. Governance challenges in global health. N Engl J Med. 2013;368(10):936-942. doi:10.1056/NEJMra1109339
  15. Hoffman SJ, Røttingen J-A. Split WHO in two: strengthening political decision-making and securing independent scientific advice. Public Health. 2014;128(2):188-194. doi:10.1016/j.puhe.2013.08.021
  16. Ruger JP. International institutional legitimacy and the World Health Organization. J Epidemiol Community Health. 2014;68(8):697-700. doi:10.1136/jech-2013-203272
  17. Schünemann HJ, Hill SR, Kakad M, et al. Transparent development of the WHO rapid advice guidelines. PLoS Med. 2007;4(5):e119.
  18. Alexander PE, Brito JP, Neumann I, et al. World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance. J Clin Epidemiol. 2016;72:98-106. doi:10.1016/j.jclinepi.2014.10.011
  19. Contandriopoulos D, Lemire M, DENIS JL, Tremblay É. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q. 2010;88(4):444-483.
  20. Dagenais C, Malo M, Robert E, Ouimet M, Berthelette D, Ridde V. Knowledge transfer on complex social interventions in public health: a scoping study. PloS One. 2013;8(12):e80233. doi:10.1371/journal.pone.0080233
  21. Lavis JN, Wilson MG, Moat KA, et al. Developing and refining the methods for a ‘one-stop shop’for research evidence about health systems. Health Res Policy Syst. 2015;13:10. doi:10.1186/1478-4505-13-10
  22. Siron S, Dagenais C, Ridde V. What research tells us about knowledge transfer strategies to improve public health in low-income countries: a scoping review. Int J Public Health. 2015;60(7):849-863.
  23. Peirson L, Catallo C, Chera S. The Registry of Knowledge Translation Methods and Tools: a resource to support evidence-informed public health. Int J Public Health. 2013;58(4):493-500.
  24. Davison CM, Ndumbe-Eyoh S, Clement C. Critical examination of knowledge to action models and implications for promoting health equity. Int J Equity Health. 2015;14:49. doi:10.1186/s12939-015-0178-7
  25. Carey G, Crammond B. Action on the social determinants of health: views from inside the policy process. Soc Sci Med. 2015;128:134-141.
  26. Smith K. Beyond Evidence Based Policy in Public Health: The Interplay of Ideas. Springer; 2013.
  27. Cairney P, Oliver K, Wellstead A. To bridge the divide between evidence and policy: reduce ambiguity as much as uncertainty. Public Adm Rev. 2016;76(3):399-402.
  28. Jewell. The Pocket Oxford Dictionary and Thesaurus. USA: Oxford University Press; 2001.
  29. Dobrow MJ, Goel V, Upshur R. Evidence-based health policy: context and utilisation. Soc Sci Med. 2004;58(1):207-217.
  30. McQueen D, Anderson LM. What counts as evidence: issues and debates. WHO Reg Publ Eur Ser. 2001;(92):63-81.
  31. Greenhalgh T. What is this knowledge that we seek to “exchange”? Milbank Q. 2010;88(4):492-499.
  32. Brownson RC, Baker EA, Leet TL, Gillespie KN, True WR. Evidence-Based Public Health. New York: Oxford University Press; 2010.
  33. Hamra GB, Laden F, Cohen AJ, Raaschou-Nielsen O, Brauer M, Loomis D. Lung cancer and exposure to nitrogen dioxide and traffic: a systematic review and meta-analysis, University of British Columbia; 2015.
  34. Medley AJ, Wong C-M, Thach TQ, Ma S, Lam T-H, Anderson HR. Cardiorespiratory and all-cause mortality after restrictions on sulphur content of fuel in Hong Kong: an intervention study. Lancet. 2002;360(9346):1646-1652.
  35. Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health interventions. J Epidemiol Community Health. 2002;56(2):119-127.
  36. Armstrong R, Pettman T, Burford B, Doyle J, Waters E. Tracking and understanding the utility of Cochrane reviews for public health decision-making. J Public Health (Oxf). 2012;34(2):309-313. doi:10.1093/pubmed/fds038
  37. World Health Organization. WHO Handbook for Guideline Development. Geneva: WHO; 2014.
  38. Easton VJ, McColl JH. STEPS Statistics Glossaryhttp://www.stats.gla.ac.uk/steps/glossary/index.html. Published 1997.
  39. Steckler A, McLeroy KR. The importance of external validity. Am J Public Health. 2008;98(1):9-10.
  40. Marmot M, Friel S, Bell R, Houweling TA, Taylor S; Commission on Social Determinants of Health. Closing the gap in a generation: health equity through action on the social determinants of health. Lancet. 2008;372(9650):1661-1669. doi:10.1016/S0140-6736(08)61690-6
  41. Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45. doi:10.1186/1471-2288-8-45
  42. WHO. Guideline: vitamin D supplementation in pregnant women.  http://www.who.int/nutrition/publications/micronutrients/guidelines/vit_d_supp_pregnant_women/en/. Published 2012.
  43. WHO. Essential nutrition actions: improving maternal, newborn, infant and young child health and nutrition. http://www.who.int/nutrition/publications/infantfeeding/essential_nutrition_actions/en/.  Published 2013. 
  44. Kramer MS, Kakuma R. Optimal duration of exclusive breastfeeding. Cochrane Database Syst Rev. 2012;(8):CD003517. doi:10.1002/14651858.CD003517.pub2
  45. WHO. Climate change and health. Published2008.
  46. Patz JA, Campbell-Lendrum D, Holloway T, Foley JA. Impact of regional climate change on human health. Nature. 2005;438(7066):310-317.
  47. Benmarhnia T, Deguen S, Kaufman JS, Smargiassi A. Vulnerability to heat-related mortality: a systematic review, meta-analysis and metaregression analysis. Epidemiology. 2015;26(6):781-793. doi:10.1097/EDE.0000000000000375
  48. Kovats RS, Hajat S. Heat stress and public health: A critical review. Annu Rev Public Health. 2008;29:41-55. doi:10.1146/annurev.publhealth.29.020907.090843
  49. Cai Z, Kuroki M, Sato T. Non‐parametric bounds on treatment effects with non‐compliance by covariate adjustment. Stat Med. 2007;26(16):3188-3204. doi:10.1002/sim.2766
  50. Ligmann-Zielinska A, Kramer DB, Cheruvelil KS, Soranno PA. Using uncertainty and sensitivity analyses in socioecological agent-based models to improve their analytical performance and policy relevance. PloS One. 2014;9(10):e109779. doi:10.1371/journal.pone.0109779
  51. Hawkins B, Parkhurst J. The ‘good governance’ of evidence in health policy.  Evid Policy. 2016;12 (4):575-592.