International Journal of Health Policy and Management

International Journal of Health Policy and Management

Making the Evaluation of Quality Indicators Count in Improving Patient Care; Comment on “Comprehensive Evaluation of Quality Indicators: Analyzing the Dutch Breast Cancer Audit”

Document Type : Commentary

Authors
1 Safety & Security Science Section, Delft University of Technology, Delft, The Netherlands
2 Centre for Safety in Healthcare, Delft University of Technology, Delft, The Netherlands
Abstract
Quality indicators are widely used, and periodic evaluation may help to ensure their usefulness in daily practice. Verheul and colleagues provide an approach for a data-driven evaluation to assess the feasibility, discriminative ability, validity, and reliability of 19 indicators used to benchmark hospital performance. They show good feasibility and discriminative ability, but also poor reliability in detecting true hospital differences which is concerning particularly for publicly reported indicators. Using their evaluation as an example, we consider in this commentary what a good evaluation should look like. Four key questions are proposed to guide choices for an evaluation approach. We highlight the importance of distinguishing between benchmarking and improvement purposes, defining the appropriate unit and criteria of evaluation, specifying thresholds for action based on the evaluation results and considering stakeholder perspectives. Aligning the evaluation approach with the intended use of indicators is critical to ensure indicators contribute to improving patient care.

Keywords

1.     Zeegers M, Veenstra GL, Gerritsen G, et al. Perceived burden due to registrations for quality monitoring and improvement in hospitals: a mixed methods study. Int J Health Policy Manag. 2022;11:183–196. doi:10.34172/ijhpm.2020.96
2.     Verheul EM, van der Linde M, Lingsma HF, et al. Comprehensive evaluation of quality indicators: analyzing the Dutch Breast Cancer Audit. Int J Health Policy Manag. 2025;14:8943. doi:10.34172/ijhpm.8943
3.     Schang L, Blotenberg I, Boywitt D. What makes a good quality indicator set? A systematic review. Int J Qual Health Care. 2021;33:1-10. doi:10.1093/intqhc/mzab107
4.     Xu D, Lewis T, Rurka M, Arling G. Approach to systematically examine the usefulness of quality measures in practice: Minnesota’s nursing home quality indicators and scoring approach. BMJ Qual Saf. 2023;32:319-329. doi:10.1136/bmjqs-2021-014384
5.     Bruinsma FFE, Nienhuijs SW, Liem RSL, Greve JWM, Marang-van de Mheen PJ. Refining the Quality Indicator Set from the Dutch Audit for Treatment of Obesity to Ensure Usefulness in Clinical Practice. Obes Surg. 2025;35:2616-2625. doi:10.1007/s11695-025-07898-2
6.     Solberg LI, Mosser G, McDonald S. The three faces of performance measurement: improvement, accountability and research. Jt Comm J Qual Improv. 1997;23:135-147. doi:10.1016/s1070-3241(16)30305-4
7.     Leis JA, Shojania KG. A primer on PDSA: executing plan-do-study-act cycles in practice, not just in name. BMJ Qual Saf. 2017;26:572-577. doi:10.1136/bmjqs-2016-006245
8.     Etchells E, Ho M, Shojania KG. Value of small sample sizes in rapid-cycle quality improvement projects. BMJ Qual Saf. 2016;25:202-206. doi:10.1136/bmjqs-2015-005094
9.     De Groot IB, Otten W, Smeets HJ, Marang-van de Mheen PJ. Is the impact of hospital performance data greater in patients who have compared hospitals? BMC Health Serv Res. 2011;11:214. doi:10.1186/1472-6963-11-214
10. Yahanda AT, Lafaro KJ, Spolverato G, Pawlik TM. A Systematic Review of the Factors that Patients Use to Choose their Surgeon. World J Surg. 2016;40:45-55. doi:10.1007/s00268-015-3246-7
11. Metcalfe D, Rios Diaz AJ, et al. Impact of public release of performance data on the behaviour of healthcare consumers and providers. Cochrane Database Syst Rev. 2018;9:CD004538. doi:10.1002/14651858.CD004538.pub3
12. Van Schie P, van Bodegom-Vos L, van Steenbergen L, Nelissen RGHH, Marang-van de Mheen PJ. Monitoring hospital performance with statistical process control after total hip and knee arthroplasty: a study to determine how much earlier worsening performance can be detected. J Bone Joint Surg Am. 2020;102:2087-2094. doi:10.2106/JBJS.20.00005
13. Kuhrij L, van Zwet E, van den Berg-Vos R, Nederkoorn P, Marang-van de Mheen PJ. Enhancing feedback on performance measures: the difference in outlier detection using a binary versus continuous outcome funnel plot and implications for quality improvement. BMJ Qual Saf. 2021;30:38-45. doi:10.1136/bmjqs-2019-009929
14. Marang-van de Mheen PJ, Vincent C. Measuring what matters: refining our approach to quality indicators. BMJ Qual Saf. 2023;32:305-308. doi:10.1136/bmjqs-2022-015221
15. Ellis RJ, Yuce TK, Hewitt DB, et al. National evaluation of patient preferences in selecting hospitals and healthcare providers. Med Care. 2020;58:867-873. doi:10.1097/MLR.0000000000001374
16. Mathes T, Pieper D, Morche J, et al. Pay for performance for hospitals. Cochrane Database Syst Rev. 2019; 2019(7):CD011156. doi:10.1002/14651858.CD011156.pub2
17. Van Schie P, van Steenbergen LN, Bodegom-Vos L, Nelissen RGHH, Marang-van de Mheen PJ. Between-hospital variation in revision rates after total hip and knee arthroplasty in the Netherlands: directing quality-improvement initiatives. J Bone Joint Surg Am. 2020;102:315-324. doi:10.2106/JBJS.19.00312
18. Franco-Santos M, Otley D. Reviewing and theorizing the unintended consequences of performance management systems. Int J Management Rev. 2018;20:696-730. doi:10.1111/ijmr.12183
19. Mannion R, Braithwaite J. Unintended consequences of performance measurement in healthcare: 20 salutary lessons from the English National Health Service. Intern Med J. 2012;42:569-574. doi:10.1111/j.1445-5994.2012.02766.x
20. Schang L, Morton A, DaSilva P, Bevan G. From data to decisions? Exploring how healthcare payers respond to the NHS Atlas of Variation in Healthcare in England. Health Policy. 2014;114:79-87. doi:10.1016/j.healthpol.2013.04.014
21. van Schie P, van Bodegom-Vos L, Zijdeman TM, Nelissen RGHH, Marang-van de Mheen PJ. Effectiveness of a multifaceted quality improvement intervention to improve patient outcomes after total hip and knee arthroplasty: a registry nested cluster randomised controlled trial. BMJ Qual Saf. 2023;32:34-46. doi:10.1136/bmjqs-2021-014472 

Articles in Press, Corrected Proof
Available Online from 22 November 2026

  • Received Date 16 October 2026
  • Revised Date 16 November 2026
  • Accepted Date 17 November 2026
  • First Published Date 22 November 2026
  • Published Date 22 November 2026