Experiences of Using Cochrane Systematic Reviews by Local HTA Units

Document Type: Short Communication


1 Department of Management, Evaluation and Health Policy, School of Public Health, University of Montreal, Montréal, QC, Canada

2 Centre de Recherche de l’Institut Universitaire en Santé Mentale de Montréal, CIUSSS de l’Est de l’Île de Montréal, Montréal, QC, Canada

3 HTA Unit, CHU de Québec – Université Laval, Québec, QC, Canada

4 Department of Multidisciplinary Services, Clinical Quality Division, CIUSSS de l’Estrie-CHUS, Sherbrooke, QC, Canada

5 HTA Unit, CHUM, Montréal, QC, Canada

6 HTA Unit, CIUSSS de la Capitale- Nationale, Québec, QC, Canada

7 HTA Unit, CIUSSS de l’Estrie – CHUS, Sherbrooke, QC, Canada

8 Department of Medicine, Faculty of Medicine and Health Science, University of Sherbrooke, Sherbrooke, QC, Canada


This study evaluated the use of Cochrane systematic reviews (CSRs) by Quebec’s local health technology assessment (HTA) units to promote efficiency in hospital decision-making. An online survey was conducted to examine: Characteristics of the HTA units; Knowledge about works and services from the Cochrane Collaboration; Level of satisfaction about the use of CSRs; Facilitating factors and barriers to the implementation of CSRs evidence in a local context; Suggestions to improve the use of CSRs. Data accuracy was checked by 2 independent evaluators. Ten HTA units participated. From their implementation a total of 321 HTA reports were published (49.8% included a SR). Works and services provided by the Cochrane collaboration were very well-known and HTA units were highly satisfied with CSRs (80%-100%). As regards to applicability in HTA and use of CSRs, major strengths were as follow: Useful as resource for search terms and background material; May reduce the workload (eg, brief review instead of full SR); Use to update a current review. Major weaknesses were: Limited use since no CSRs were available for many HTA projects; Difficulty to apply findings to local context; Focused only on efficacy and innocuity; Cannot be used as a substitute to a full HTA report. This study provided a unique context of assessment with a familiar group of producers, users and disseminators of CSRs in hospital setting. Since they generally used other articles from the literature or produce an original SR in complement with CSRs, this led to suggestions to improve their use of CSRs. However, the main limit for the use of CRS in local HTA will remain its lack of contextualisation. As such, this study reinforces the need to consider the notion of complementarity of experimental data informing us about causality and contextual data, allowing decision-making adapted to local issues.


  1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312(7023):71-72. doi:10.1136/bmj.312.7023.71
  2. Poder TG, Bellemare CA. Importance of contextual data in producing health technology assessment recommendations: a case study. Int J Technol Assess Health Care. 2018;34(1):63-67. doi:10.1017/s0266462317004469
  3. About Cochrane Reviews. Cochrane Library website. https://www.cochranelibrary.com/about/about-cochrane-reviews.  Accessed November15, 2018.
  4. Green S, Higgins J, Alderson P, Clarke M, Mulrow C, Oxman A. Chapter 1: Introduction. In: Higgins J, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. West Sussex, England: The Cochrane Collaboration and John Wiley; 2008.
  5. Tricco AC, Tetzlaff J, Moher D. Knowledge synthesis. In: Straus SE, Tetroe J, Graham ID, eds. Knowledge Translation in Health Care. Wiley-Blackwell; 2013:29-49. doi:10.1002/9781118413555.ch04
  6. Marquez C, Johnson AM, Jassemi S, et al. Enhancing the uptake of systematic reviews of effects: what is the best format for health care managers and policy-makers? a mixed-methods study. Implement Sci. 2018;13(1):84. doi:10.1186/s13012-018-0779-9
  7. Wallace J, Byrne C, Clarke M. Making evidence more wanted: a systematic review of facilitators to enhance the uptake of evidence from systematic reviews and meta-analyses. Int J Evid Based Healthc. 2012;10(4):338-346. doi:10.1111/j.1744-1609.2012.00288.x
  8. Dobbins M, DeCorby K, Twiddy T. A knowledge transfer strategy for public health decision makers. Worldviews Evid Based Nurs. 2004;1(2):120-128. doi:10.1111/j.1741-6787.2004.t01-1-04009.x
  9. Dobbins M, Cockerill R, Barnsley J. Factors affecting the utilization of systematic reviews. A study of public health decision makers. Int J Technol Assess Health Care. 2001;17(2):203-214. doi:10.1017/s0266462300105069
  10. Perrier L, Mrklas K, Lavis JN, Straus SE. Interventions encouraging the use of systematic reviews by health policymakers and managers: a systematic review. Implement Sci. 2011;6:43. doi:10.1186/1748-5908-6-43
  11. Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10 Suppl 1:35-48. doi:10.1258/1355819054308549
  12. Lavis JN. How can we support the use of systematic reviews in policymaking? PLoS Med. 2009;6(11):e1000141. doi:10.1371/journal.pmed.1000141
  13. Alper BS, Hand JA, Elliott SG, et al. How much effort is needed to keep up with the literature relevant for primary care? J Med Libr Assoc. 2004;92(4):429-437.
  14. Velasco M, Perleth M, Drummond M, et al. Best practice in undertaking and reporting health technology assessments. Working group 4 report. Int J Technol Assess Health Care. 2002;18(2):361-422. doi:10.1017/s0266462302000284
  15. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. doi:10.1371/journal.pmed.1000097
  16. Kung J, Chiappelli F, Cajulis OO, et al. From systematic reviews to clinical recommendations for evidence-based health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J. 2010;4:84-91. doi:10.2174/1874210601004020084
  17. Whiting P, Savović J, Higgins JP, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225-234. doi:10.1016/j.jclinepi.2015.06.005
  18. Higgins JP, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. doi:10.1136/bmj.d5928
  19. Kringos DS, Sunol R, Wagner C, et al. The influence of context on the effectiveness of hospital quality improvement strategies: a review of systematic reviews. BMC Health Serv Res. 2015;15:277. doi:10.1186/s12913-015-0906-0
  20. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005;10(1):45-53. doi:10.1177/135581960501000110
  21. Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy. 2005;10 Suppl 1:6-20. doi:10.1258/1355819054308576
  22. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581-629. doi:10.1111/j.0887-378X.2004.00325.x
  23. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10 Suppl 1:21-34. doi:10.1258/1355819054308530
  24. Kongsted HC, Konnerup M. Are more observational studies being included in Cochrane Reviews? BMC Res Notes. 2012;5:570. doi:10.1186/1756-0500-5-570
  25. Bunn F, Trivedi D, Alderson P, Hamilton L, Martin A, Iliffe S. The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research. Syst Rev. 2014;3:125. doi:10.1186/2046-4053-3-125
  26. Davis J, Mengersen K, Bennett S, Mazerolle L. Viewing systematic reviews and meta-analysis in social research through different lenses. Springerplus. 2014;3:511. doi:10.1186/2193-1801-3-511
  27. Kelly E. Systematic and just: the use of a systematic review methodology in social work research. Soc Work Soc Sci Rev. 2015;15(3):72-85.
  28. Rutter D, Francis J, Coren E, Fisher M. SCIE systematic Research Reviews: Guidelines. 2nd ed. Social Care Institute for Excellence (SCIE); 2011.
  29. Pedersen VH, Dagenais P, Lehoux P. Multi-source synthesis of data to inform health policy. Int J Technol Assess Health Care. 2011;27(3):238-246. doi:10.1017/s0266462311000213
  30. Anglemyer A, Horvath HT, Bero L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst Rev. 2014(4):MR000034. doi:10.1002/14651858.MR000034.pub2
  31. Munthe-Kaas H, Nøkleby H, Nguyen L. Systematic mapping of checklists for assessing transferability. Syst Rev. 2019;8(1):22. doi:10.1186/s13643-018-0893-4
  32. Poder TG. Using the health technology assessment toolbox to facilitate procurement: the case of smart pumps in a Canadian hospital. Int J Technol Assess Health Care. 2017;33(1):54-62. doi:10.1017/s0266462317000125
  33. Poder TG, Godbout ST, Bellemare C. Dual vs. single computer monitor in a Canadian hospital Archiving Department: a study of efficiency and satisfaction. Health Inf Manag. 2011;40(3):20-25. doi:10.1177/183335831104000303
  34. Poder TG, Pruneau D, Dorval J, et al. Pressure infusion cuff and blood warmer during massive transfusion: an experimental study about hemolysis and hypothermia. PLoS One. 2016;11(10):e0163429. doi:10.1371/journal.pone.0163429
  35. Chen Z, Deblois S, Toporowicz K, et al. Yield of CT pulmonary angiography in the diagnosis of acute pulmonary embolism: short report. BMC Res Notes. 2019;12(1):41. doi:10.1186/s13104-019-4076-8