International Journal of Health Policy and Management

International Journal of Health Policy and Management

Measuring Quality or Measuring Noise? Lessons From the Dutch Breast Cancer Audit; Comment on “Comprehensive Evaluation of Quality Indicators: Analyzing the Dutch Breast Cancer Audit”

Articles in Press

Document Type : Commentary

Author
Olaf Johan Hartmann
Department of Breast and Endocrine Surgery, Akershus University Hospital, Lørenskog, Norway
Abstract
Cancer registries and clinical audits are increasingly used to compare healthcare performance across institutions. However, observed differences between hospitals are often interpreted as true differences in quality without considering whether the indicators can reliably distinguish between providers. This commentary discusses the framework applied by Verheul and colleagues to the Dutch Breast Cancer Audit. It distinguishes registry data quality from indicator performance and describes four complementary properties in sequence: feasibility, discriminative ability, validity, and reliability, with rankability as a measure of the latter. Reliability deserves particular attention when indicators guide public reporting, reimbursement, policy, or patient choice, because low rankability indicates that apparent differences may largely reflect random variation. International standards and the broader reliability literature support pre-use testing, transparent reporting of uncertainty and sample-size requirements, periodic re-evaluation, and restriction or retirement of unreliable indicators. Without adequate reliability, institutional differences should not be interpreted as true differences in quality of care.
Keywords
Quality Indicators
Clinical Registries
Reliability
Rankability
Benchmarking
Netherlands
1.     Verheul EM, van der Linde M, Lingsma HF, et al. Comprehensive evaluation of quality indicators: analyzing the Dutch Breast Cancer Audit. Int J Health Policy Manag. 2025;14:8943. doi:10.34172/ijhpm.8943
2.     Vos EL, Lingsma HF, Jager A, et al. Effect of case-mix and random variation on breast cancer care quality indicators and their rankability. Value Health. 2020;23(9):1191-1199. doi:10.1016/j.jval.2019.12.014
3.     Donabedian A. The quality of care. How can it be assessed? JAMA. 1988;260(12):1743-1748. doi:10.1001/jama.260.12.1743
4.     Maes-Carballo M, Gómez-Fandiño Y, Reinoso-Hermida A, et al. Quality indicators for breast cancer care: a systematic review. Breast. 2021;59:221-231. doi:10.1016/j.breast.2021.06.013
5.     Backes J, Vogel J, Geissler A, et al. Welcome to the jungle: collection and evaluation of quality indicators with the QUALICATOR instrument. BMJ Qual Saf. 2026;35:565-575. doi:10.1136/bmjqs-2026-020265
6.     Jones P, Shepherd M, Wells S, Le Fevre J, Ameratunga S. Review article: what makes a good healthcare quality indicator? A systematic review and validation study. Emerg Med Australas. 2014;26(2):113-124. doi:10.1111/1742-6723.12195
7.     Partnership for Quality Measurement. What endorsement means. https://p4qm.org/em/about. Accessed September 2, 2026.
8.     Glance LG, Joynt Maddox K, Johnson K, et al. National Quality Forum guidelines for evaluating the scientific acceptability of risk-adjusted clinical outcome measures: a report from the National Quality Forum Scientific Methods Panel. Ann Surg. 2020;271(6):1048-1055. doi:10.1097/SLA.0000000000003592
9.     Shih T, Dimick JB. Reliability of readmission rates as a hospital quality measure in cardiac surgery. Ann Thorac Surg. 2014;97(4):1214-1218. doi:10.1016/j.athoracsur.2013.11.048
10. Nieser KJ, Harris AHS. Comparing methods for assessing the reliability of health care quality measures. Stat Med. 2024;43(23):4575-4594. doi:10.1002/sim.10197
11. Adams JL, Mehrotra A, McGlynn EA. Estimating Reliability and Misclassification in Physician Profiling. RAND Corporation; 2010. TR-863-MMS. https://www.rand.org/pubs/technical_reports/TR863.html.
12. Schang L, Blotenberg I, Boywitt D. What makes a good quality indicator set? A systematic review of criteria. Int J Qual Health Care. 2021;33(3):mzab107. doi:10.1093/intqhc/mzab107
13. Meyer GS, Nelson EC, Pryor DB, et al. More quality measures versus measuring what matters: a call for balance and parsimony. BMJ Qual Saf. 2012;21(11):964-968. doi:10.1136/bmjqs-2012-001081
International Journal of Health Policy and Management

Articles in Press, Corrected Proof
Available Online from 29 September 2026

  • Received Date 29 July 2026
  • Revised Date 07 September 2026
  • Accepted Date 10 September 2026
  • First Published Date 29 September 2026
  • Published Date 29 September 2026