Common Features of Selection Processes of Health System Performance Indicators in Primary Healthcare: A Systematic Review

Document Type : Review Article

Authors

1 National Centre for Epidemiology and Population Health, Australian National University, Canberra, ACT, Australia

2 School of Public Health and Community Medicine, University of New South Wales, Sydney, NSW, Australia

3 Menzies School of Health Research, Brisbane, QLD, Australia

Abstract

Background 
Health system performance indicators are widely used to assess primary healthcare (PHC) performance. Despite the numerous tools and some convergence on indicator criteria, there is not a clear understanding of the common features of indicator selection processes. We aimed to review the literature to identify papers that document indicator selection processes for health system performance indicators in PHC.

Methods 
We searched the online databases Scopus, Medline, and CINAHL, as well as the grey literature, without time restrictions, initially on July 31, 2019 followed by an update November 13, 2020. Empirical studies or reports were included if they described the selection of health system performance indicators or frameworks, that included PHC indicators. A combination of the process focussed research question and qualitative analysis meant a quality appraisal tool or assessment of bias could not meaningfully be applied to assess individual studies. We undertook an inductive analysis based on potential indicator selection processes criteria, drawn from health system performance indicator appraisal tools reported in the literature.

Results 
We identified 16 503 records of which 28 were included in the review. Most studies used a descriptive case study design. We found no consistent variations between indicator selection processes of health systems of high income and low- or lower-middle income countries. Identified common features of selection processes for indicators in PHC include literature review or adaption of an existing framework as an initial step; a consensus building process with stakeholders; structuring indicators into categories; and indicator criteria focusing on validity and feasibility. The evidence around field testing with utility and consideration of reporting burden was less clear.

Conclusion 
Our findings highlight several characteristics of health system indicator selection processes. These features provide the groundwork to better understand how to value indicator selection processes in PHC.

Keywords

References

  1. European Union. Expert Panel on effective ways of investing in Health (EXPH): Preliminary report on Tools and Methodologies for Assessing the Performance of Primary Care. 2017. https://ec.europa.eu/health/sites/health/files/expert_panel/docs/017_assessing_performance_primarycare_en.pdf.
  2. Levesque JF, Sutherland K. Combining patient, clinical and system perspectives in assessing performance in healthcare: an integrated measurement framework. BMC Health Serv Res. 2020;20(1):23. doi:1186/s12913-019-4807-5
  3. Hirschhorn LR, Baynes C, Sherr K, et al. Approaches to ensuring and improving quality in the context of health system strengthening: a cross-site analysis of the five African Health Initiative Partnership programs. BMC Health Serv Res. 2013;13 Suppl 2:S8. doi:1186/1472-6963-13-s2-s8
  4. Perić N, Hofmarcher MM, Simon J. Headline indicators for monitoring the performance of health systems: findings from the european Health Systems_Indicator (euHS_I) survey. Arch Public Health. 2018;76:32. doi:1186/s13690-018-0278-0
  5. Braithwaite J, Hibbert P, Blakely B, et al. Health system frameworks and performance indicators in eight countries: a comparative international analysis. SAGE Open Med. 2017;5:2050312116686516. doi:1177/2050312116686516
  6. Marshall MN, Shekelle PG, McGlynn EA, Campbell S, Brook RH, Roland MO. Can health care quality indicators be transferred between countries? Qual Saf Health Care. 2003;12(1):8-12. doi:1136/qhc.12.1.8
  7. Noto G, Corazza I, Kļaviņa K, Lepiksone J, Nuti S. Health system performance assessment in small countries: the case study of Latvia. Int J Health Plann Manage. 2019;34(4):1408-1422. doi:1002/hpm.2803
  8. Fekri O, Macarayan ER, Klazinga N. Health System Performance Assessment in the WHO European Region: Which Domains and Indicators have been Used by Member States for its Measurement? Health Evidence Network synthesis report 55. 2018. https://www.euro.who.int/en/publications/abstracts/health-system-performance-assessment-in-the-who-european-region-which-domains-and-indicators-have-been-used-by-member-states-for-its-measurement-2018.
  9. World Health Organization. Monitoring the building blocks of health systems: a handbook of indicators and their measurement strategies. 2010. https://www.who.int/healthinfo/systems/WHO_MBHSS_2010_full_web.pdf.
  10. de Koning J, Smulders A, Klazinga N. Appraisal of Indicators through Research and Evaluation (AIRE). Amsterdam: Academic Medical Center- University of Amsterdam; 2007.
  11. Reiter A, Fischer B, Kötting J, et al. QUALIFY: Instrument for the Assessment of Quality Indicators. 2007. https://www.researchgate.net/publication/267256474_QUALIFY_Instrument_for_the_Assessment_of_Quality_Indicators.
  12. Barbazza E, Klazinga NS, Kringos DS. Exploring the actionability of healthcare performance indicators for quality of care: a qualitative analysis of the literature, expert opinion and user experience. BMJ Qual Saf. 2021;30(12):1010-1020. doi:1136/bmjqs-2020-011247
  13. Ahuja S, Gronholm PC, Shidhaye R, Jordans M, Thornicroft G. Development of mental health indicators at the district level in Madhya Pradesh, India: mixed methods study. BMC Health Serv Res. 2018;18(1):867. doi:1186/s12913-018-3695-4
  14. Ebert ST, Pittet V, Cornuz J, Senn N. Development of a monitoring instrument to assess the performance of the Swiss primary care system. BMC Health Serv Res. 2017;17(1):789. doi:1186/s12913-017-2696-z
  15. Fukuma S, Shimizu S, Niihata K, et al. Development of quality indicators for care of chronic kidney disease in the primary care setting using electronic health data: a RAND-modified Delphi method. Clin Exp Nephrol. 2017;21(2):247-256. doi:1007/s10157-016-1274-8
  16. Lee B, Park SY. Developing key performance indicators for guaranteeing right to health and access to medical service for persons with disabilities in Korea: using a modified Delphi. PLoS One. 2018;13(12):e0208651. doi:1371/journal.pone.0208651
  17. O'Donnell S, Doyle G, O'Malley G, et al. Establishing consensus on key public health indicators for the monitoring and evaluating childhood obesity interventions: a Delphi panel study. BMC Public Health. 2020;20(1):1733. doi:1186/s12889-020-09814-y
  18. Ramalho A, Castro P, Gonçalves-Pinho M, et al. Primary health care quality indicators: an umbrella review. PLoS One. 2019;14(8):e0220888. doi:1371/journal.pone.0220888
  19. Blozik E, Nothacker M, Bunk T, Szecsenyi J, Ollenschläger G, Scherer M. Simultaneous development of guidelines and quality indicators -- how do guideline groups act? A worldwide survey. Int J Health Care Qual Assur. 2012;25(8):712-729. doi:1108/09526861211270659
  20. Klazinga N, Fischer C, ten Asbroek A. Health services research related to performance indicators and benchmarking in Europe. J Health Serv Res Policy. 2011;16 Suppl 2:38-47. doi:1258/jhsrp.2011.011042
  21. Schang L, Blotenberg I, Boywitt D. What makes a good quality indicator set? A systematic review of criteria. Int J Qual Health Care. 2021;33(3):mzab107. doi:1093/intqhc/mzab107
  22. World Health Organization. Primary Health Care (PHC). 2019. https://www.who.int/primary-health/en/.
  23. World Health Organization. Declaration on Primary Health Care: Astana, 2018. 2019. https://www.who.int/primary-health/conference-phc/declaration.
  24. World Health Organization. The world health report 2008: primary health care now more than ever. 2008. https://www.who.int/whr/2008/whr08_en.pdf.
  25. Pettigrew LM, De Maeseneer J, Anderson MI, Essuman A, Kidd MR, Haines A. Primary health care and the Sustainable Development Goals. Lancet. 2015;386(10009):2119-2121. doi:1016/s0140-6736(15)00949-6
  26. Starfield B. Primary care: an increasingly important contributor to effectiveness, equity, and efficiency of health services. SESPAS report 2012. Gac Sanit. 2012;26 Suppl 1:20-26. doi:1016/j.gaceta.2011.10.009
  27. Kringos DS, Boerma WG, Hutchinson A, van der Zee J, Groenewegen PP. The breadth of primary care: a systematic literature review of its core dimensions. BMC Health Serv Res. 2010;10:65. doi:1186/1472-6963-10-65
  28. Kringos DS, Boerma WG, Bourgueil Y, et al. The European primary care monitor: structure, process and outcome indicators. BMC Fam Pract. 2010;11:81. doi:1186/1471-2296-11-81
  29. Primary Health Care Performance Initiative. Measuring Primary Health Care Performance. 2018. https://improvingphc.org/measuring-primary-health-care-performance.
  30. Shi L, Masís DP, Guanais FC. Measurement of Primary Care: The Johns Hopkins Primary Care Assessment Tool 2012. https://www.jhsph.edu/research/centers-and-institutes/johns-hopkins-primary-care-policy-center/pca_tools.html.
  31. United Nations. Sustainable Development Goals Knowledge Platform. https://sustainabledevelopment.un.org/sdgs.
  32. Fracolli LA, Gomes MF, Nabão FR, Santos MS, Cappellini VK, de Almeida AC. Primary health care assessment tools: a literature review and metasynthesis. Cien Saude Colet. 2014;19(12):4851-4860. doi:1590/1413-812320141912.00572014
  33. Bangalore Sathyananda R, de Rijk A, Manjunath U, Krumeich A, van Schayck CP. Primary health Centres' performance assessment measures in developing countries: review of the empirical literature. BMC Health Serv Res. 2018;18(1):627. doi:1186/s12913-018-3423-0
  34. European Union. A New Drive for Primary Care in Europe: Rethinking the Assessment Tools and Methodologies. Report of the Expert Group on Health Systems Performance Assessment. 2018. https://ec.europa.eu/health/sites/health/files/systems_performance_assessment/docs/2018_primarycare_eg_en.pdf.
  35. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. doi:1371/journal.pmed.1000097
  36. Perera R, Dowell T, Crampton P, Kearns R. Panning for gold: an evidence-based tool for assessment of performance indicators in primary health care. Health Policy. 2007;80(2):314-327. doi:1016/j.healthpol.2006.03.011
  37. The World Bank. World Bank Country and Lending Groups. 2020. https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups.
  38. Aller MB, Vargas I, Coderch J, et al. Development and testing of indicators to measure coordination of clinical information and management across levels of care. BMC Health Serv Res. 2015;15:323. doi:1186/s12913-015-0968-z
  39. Barbazza E, Kringos D, Kruse I, Klazinga NS, Tello JE. Creating performance intelligence for primary health care strengthening in Europe. BMC Health Serv Res. 2019;19(1):1006. doi:1186/s12913-019-4853-z
  40. Blozik E, Reich O, Rapold R, Scherer M. Evidence-based indicators for the measurement of quality of primary care using health insurance claims data in Switzerland: results of a pragmatic consensus process. BMC Health Serv Res. 2018;18(1):743. doi:1186/s12913-018-3477-z
  41. Campbell SM, Kontopantelis E, Hannon K, Burke M, Barber A, Lester HE. Framework and indicator testing protocol for developing and piloting quality indicators for the UK quality and outcomes framework. BMC Fam Pract. 2011;12:85. doi:1186/1471-2296-12-85
  42. Carinci F, Van Gool K, Mainz J, et al. Towards actionable international comparisons of health system performance: expert revision of the OECD framework and quality indicators. Int J Qual Health Care. 2015;27(2):137-146. doi:1093/intqhc/mzv004
  43. Claessen SJ, Francke AL, Belarbi HE, Pasman HR, van der Putten MJ, Deliens L. A new set of quality indicators for palliative care: process and results of the development trajectory. J Pain Symptom Manage. 2011;42(2):169-182. doi:1016/j.jpainsymman.2010.10.267
  44. Coma E, Ferran M, Méndez L, Iglesias B, Fina F, Medina M. Creation of a synthetic indicator of quality of care as a clinical management standard in primary care. Springerplus. 2013;2(1):51. doi:1186/2193-1801-2-51
  45. Cookson R, Asaria M, Ali S. Health Equity Indicators for the English NHS: a longitudinal whole-population study at the small-area level. 2016. https://www.ncbi.nlm.nih.gov/books/NBK385236/.
  46. De Bie J, Kijlstra NB, Daemen BJ, Bouvy ML. The development of quality indicators for community pharmacy care. BMJ Qual Saf. 2011;20(8):666-671. doi:1136/bmjqs.2010.045237
  47. Engels Y, Dautzenberg M, Campbell S, et al. Testing a European set of indicators for the evaluation of the management of primary care practices. Fam Pract. 2006;23(1):137-147. doi:1093/fampra/cmi091
  48. Leemans K, Cohen J, Francke AL, et al. Towards a standardized method of developing quality indicators for palliative care: protocol of the quality indicators for palliative care (Q-PAC) study. BMC Palliat Care. 2013;12:6. doi:1186/1472-684x-12-6
  49. Parker D, Wensing M, Esmail A, Valderas JM. Measurement tools and process indicators of patient safety culture in primary care. A mixed methods study by the LINNEAUS collaboration on patient safety in primary care. Eur J Gen Pract. 2015;21 Suppl 1:26-30. doi:3109/13814788.2015.1043732
  50. ni Riain A, Vahey C, Kennedy C, Campbell S, Collins C. Roadmap for developing a national quality indicator set for general practice. Int J Health Care Qual Assur. 2015;28(4):382-393. doi:1108/ijhcqa-09-2014-0091
  51. Rushforth B, Stokes T, Andrews E, et al. Developing 'high impact' guideline-based quality indicators for UK primary care: a multi-stage consensus process. BMC Fam Pract. 2015;16:156. doi:1186/s12875-015-0350-6
  52. Stanciu MA, Law RJ, Myres P, et al. The development of the Primary Care Clusters Multidimensional Assessment (PCCMA): a mixed-methods study. Health Policy. 2020;124(2):152-163. doi:1016/j.healthpol.2019.12.004
  53. Barrett TJ, Bartsch DA, Zahniser JH, Belanger S. Implementing and evaluating outcome indicators of performance for mental health agencies. J Healthc Qual. 1998;20(3):6-13. doi:1111/j.1945-1474.1998.tb00254.x
  54. Herndon JB, Crall JJ, Aravamudhan K, et al. Developing and testing pediatric oral healthcare quality measures. J Public Health Dent. 2015;75(3):191-201. doi:1111/jphd.12087
  55. Reedy AM, Luna RG, Olivas GS, Sujeer A. Local public health performance measurement: implementation strategies and lessons learned from aligning program evaluation indicators with the 10 essential public health services. J Public Health Manag Pract. 2005;11(4):317-325. doi:1097/00124784-200507000-00010
  56. Hutchison B, Haj-Ali W, Dobell G, Yeritsyan N, Degani N, Gushue S. Prioritizing and implementing primary care performance measures for Ontario. Healthc Policy. 2020;16(1):43-57. doi:12927/hcpol.2020.26291
  57. Katz A, Soodeen RA, Bogdanovic B, De Coster C, Chateau D. Can the quality of care in family practice be measured using administrative data? Health Serv Res. 2006;41(6):2238-2254. doi:1111/j.1475-6773.2006.00589.x
  58. Terner M, D'Silva J, Tipper B, Krylova O, Webster G. Assessing primary healthcare using pan- Canadian indicators of health and health system performance. Healthc Q. 2013;16(2):9-12.
  59. Australian Institute of Health and Welfare (AIHW). Towards national indicators of safety and quality in health care. 2009. https://www.aihw.gov.au/getmedia/a143a228-0e9e-4098-bea6-a84053446bbc/hse-75-10792_c02.pdf.aspx.
  60. Gribben B, Coster G, Pringle M, Simon J. Quality of care indicators for population-based primary care in New Zealand. N Z Med J. 2002;115(1151):163-166.
  61. Wong ST, Yin D, Bhattacharyya O, Wang B, Liu L, Chen B. Developing a performance measurement framework and indicators for community health service facilities in urban China. BMC Fam Pract. 2010;11:91. doi:1186/1471-2296-11-91
  62. Nambiar D, Sankar DH, Negi J, Nair A, Sadanandan R. Monitoring universal health coverage reforms in primary health care facilities: creating a framework, selecting and field-testing indicators in Kerala, India. PLoS One. 2020;15(8):e0236169. doi:1371/journal.pone.0236169
  63. Prytherch H, Nafula M, Kandie C, et al. Quality management: where is the evidence? Developing an indicator-based approach in Kenya. Int J Qual Health Care. 2017;29(1):19-25. doi:1093/intqhc/mzw147
  64. Sarriot E, Ricca J, Ryan L, Basnet J, Arscott-Mills S. Measuring sustainability as a programming tool for health sector investments: report from a pilot sustainability assessment in five Nepalese health districts. Int J Health Plann Manage. 2009;24(4):326-350. doi:1002/hpm.1012
  65. Veillard J, Cowling K, Bitton A, et al. Better measurement for performance improvement in low- and middle-income countries: the primary health care performance initiative (PHCPI) experience of conceptual framework development and indicator selection. Milbank Q. 2017;95(4):836-883. doi:1111/1468-0009.12301
  66. Baxter P, Jack S. Qualitative case study methodology: study design and implementation for novice researchers. Qual Rep. 2010;13(4):544-559. doi:46743/2160-3715/2008.1573
  67. de Bruin-Kooistra M, Amelink-Verburg MP, Buitendijk SE, Westert GP. Finding the right indicators for assessing quality midwifery care. Int J Qual Health Care. 2012;24(3):301-310. doi:1093/intqhc/mzs006
  68. Perera R, Dowell A, Crampton P. Painting by numbers: a guide for systematically developing indicators of performance at any level of health care. Health Policy. 2012;108(1):49-59. doi:1016/j.healthpol.2012.07.008
  69. Mainz J. Developing evidence-based clinical indicators: a state of the art methods primer. Int J Qual Health Care. 2003;15 Suppl 1:i5-11. doi:1093/intqhc/mzg084
  70. Kötter T, Blozik E, Scherer M. Methods for the guideline-based development of quality indicators--a systematic review. Implement Sci. 2012;7:21. doi:1186/1748-5908-7-21
  71. Campbell SM, Braspenning J, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care. 2002;11(4):358-364. doi:1136/qhc.11.4.358
  72. Jandhyala R. Delphi, non-RAND modified Delphi, RAND/UCLA appropriateness method and a novel group awareness and consensus methodology for consensus measurement: a systematic literature review. Curr Med Res Opin. 2020;36(11):1873-1887. doi:1080/03007995.2020.1816946
  73. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476. doi:1371/journal.pone.0020476
  74. Barson S, Doolan-Noble F, Gray J, Gauld R. Healthcare leaders' views on successful quality improvement initiatives and context. J Health Organ Manag. 2017;31(1):54-63. doi:1108/jhom-10-2016-0191
  75. Crampton P, Perera R, Crengle S, et al. What makes a good performance indicator? Devising primary care performance indicators for New Zealand. N Z Med J. 2004;117(1191):U820.
  76. Freeman T. Using performance indicators to improve health care quality in the public sector: a review of the literature. Health Serv Manage Res. 2002;15(2):126-137. doi:1258/0951484021912897
  77. Klazinga N, Stronks K, Delnoij D, Verhoeff A. Indicators without a cause. Reflections on the development and use of indicators in health care from a public health perspective. Int J Qual Health Care. 2001;13(6):433-438. doi:1093/intqhc/13.6.433
  78. Kötter T, Schaefer FA, Scherer M, Blozik E. Involving patients in quality indicator development - a systematic review. Patient Prefer Adherence. 2013;7:259-268. doi:2147/ppa.s39803
  79. Santana MJ, Ahmed S, Lorenzetti D, et al. Measuring patient-centred system performance: a scoping review of patient-centred care quality indicators. BMJ Open. 2019;9(1):e023596. doi:1136/bmjopen-2018-023596
  80. Hilarion P, Suñol R, Groene O, Vallejo P, Herrera E, Saura RM. Making performance indicators work: the experience of using consensus indicators for external assessment of health and social services at regional level in Spain. Health Policy. 2009;90(1):94-103. doi:1016/j.healthpol.2008.08.002
  81. Oostra DL, Nieuwboer MS, Olde Rikkert MGM, Perry M. Development and pilot testing of quality improvement indicators for integrated primary dementia care. BMJ Open Qual. 2020;9(2):e000916.. doi:1136/bmjoq-2020-000916
  82. Saturno-Hernández PJ, Fernández-Elorriaga M, Martínez-Nicolás I, Poblano-Verástegui O. Construction and pilot test of a set of indicators to assess the implementation and effectiveness of the who safe childbirth checklist. BMC Pregnancy Childbirth. 2018;18(1):154. doi:1186/s12884-018-1797-y
  83. NHS Improvement. A model for measuring quality care. https://improvement.nhs.uk/documents/2135/measuring-quality-care-model.pdf.
  84. Hyppönen H, Ronchi E, Adler-Milstein J. Health care performance indicators for health information systems. Stud Health Technol Inform. 2016;222:181-194.
  85. Kidd MR, Anderson MI, Obazee EM, Prasad PN, Pettigrew LM. The need for global primary care development indicators. Lancet. 2015;386(9995):737. doi:1016/s0140-6736(15)61532-x
  86. Ham C, Raleigh V, Foot C, Robertson R, Alderwick H. Measuring the performance of local health systems: a review for the Department of Health. 2015. https://www.kingsfund.org.uk/sites/default/files/field/field_publication_file/measuring-the-performance-of-local-health-systems-dh-review-kingsfund-oct15.pdf.
  87. Hong QN, Pluye P, Fabregues S, et al. Mixed Methods Appraisal Tool (MMAT) Version 2018. 2018. http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/MMAT_2018_criteria-manual_2018-08-01_ENG.pdf.
  88. Franklin C, Zhang A, Froerer A, Johnson S. Solution focused brief therapy: a systematic review and meta-summary of process research. J Marital Fam Ther. 2017;43(1):16-30. doi:1111/jmft.12193
  89. Carroll C, Booth A. Quality assessment of qualitative evidence for systematic review and synthesis: is it meaningful, and if so, how should it be performed? Res Synth Methods. 2015;6(2):149-154. doi:1002/jrsm.1128
  90. Whiting P, Wolff R, Mallett S, Simera I, Savović J. A proposed framework for developing quality assessment tools. Syst Rev. 2017;6(1):204. doi:1186/s13643-017-0604-6
  91. Davidsson P, Gruenhagen JH. Fulfilling the process promise: a review and agenda for new venture creation process research. Entrep Theory Pract. 2021;45(5):1083-1118. doi:1177/1042258720930991
  92. Berends H, Deken F. Composing qualitative process research. Strateg Organ. 2021;19(1):134-146. doi:1177/1476127018824838
  93. Hudson J, Kühner S. Qualitative comparative analysis and applied public policy analysis: new applications of innovative methods. Policy Soc. 2013;32(4):279-287. doi:1016/j.polsoc.2013.10.001
  94. White H, Sabarwal S. Quasi-experimental Design and Methods, Methodological Briefs: Impact Evaluation 8. 2014. https://www.unicef-irc.org/KM/IE/img/downloads/Quasi-Experimental_Design_and_Methods_ENG.pdf.
  95. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277-1288. doi:1177/1049732305276687
International Journal of Health Policy and Management
Volume 11, Issue 12
December 2022
Pages 2805-2815

Files

History

  • Receive Date: 15 April 2021
  • Revise Date: 25 February 2022
  • Accept Date: 06 March 2022
  • First Publish Date: 07 March 2022

Share

How to cite

Statistics