Kerman University of Medical Sciences International Journal of Health Policy and Management 2322-5939 14 1 2025 12 01 Comprehensive Evaluation of Quality Indicators: Analyzing the Dutch Breast Cancer Audit 1 12 4800 10.34172/ijhpm.8943 EN Elfi M. Verheul Center for Medical Decision Making, Department of Public Health, Erasmus University Medical Center, Rotterdam, The Netherlands Dutch Institute for Clinical Auditing, Leiden, The Netherlands Margrietha Van Der Linde Center for Medical Decision Making, Department of Public Health, Erasmus University Medical Center, Rotterdam, The Netherlands Hester F. Lingsma Center for Medical Decision Making, Department of Public Health, Erasmus University Medical Center, Rotterdam, The Netherlands Elvira Vos Department of Surgery, Rhode Island Hospital, Warren Alpert Medical School of Brown University, Providence, RI, USA Sabine Siesling Department of Research, Netherlands Comprehensive Cancer Organization (IKNL), Utrecht, The Netherlands Department of Health Technology and Services Research, Technical Medical Centre, University of Twente, Enschede, The Netherlands Linetta B. Koppert Department of Surgery, Erasmus MC Cancer Institute, Rotterdam, The Netherlands 0000-0003-1944-6777 NBCA Consortium# # A full list of the collaborators of the NBCA Consortium is provided at the end of the article Journal Article 2024 10 21 Background Quality indicators (QIs) are widely used to benchmark hospital performance and improve quality of care but are often based on expert opinion rather than data-driven assessment. This study aims to evaluate QIs, using a framework that assesses four criteria: Feasibility, discriminative ability, validity, and reliability. Methods We used data from the Dutch breast cancer registry (NABON Breast Cancer Audit, NBCA) and included all surgically treated breast cancer patients in the Netherlands between 2021-2023. Eighteen QIs were evaluated. Feasibility was determined by QI numerator completeness, with >90% data availability considered feasible. Discriminative ability was assessed by between-hospital variation in QI scores, where an interquartile range (IQR) >10% indicated good discrimination. Validity was evaluated by the impact of case-mix adjustment and considered low when (pseudo-) R² <0.10. The (pseudo-)R² reflects the proportion of variance in QI scores explained by all case-mix variables in a regression model. Reliability was assessed by rankability, the proportion of between-hospital variation not due to chance and, therefore, explainable by quality of care. Rankability >75% was considered high. Results After exclusion of one QI with feasibility <1%, feasibility ranged from 80.2%-100%, and 15 QIs were feasible. Overall, the IQR ranged from 1 to 36, with 8 QIs having an IQR higher than 10, indicating good discriminative ability. The (pseudo-)R² ranged from 0.01-0.53, with 11 QIs showing low case-mix impact. Rankability ranged from 0-69%, with none of the QIs having a high rankability. None of the QIs met all preset criteria, but six QIs met at least three out of four criteria. Conclusion The QIs employed by the Dutch breast cancer registry fulfilled most criteria, but rankability is a concern and requires specific attention, especially for public reporting. Our results show the importance of considering feasibility, discriminative ability, validity, and reliability when evaluating QIs, and these should also be taken into account when developing new QIs.

https://www.ijhpm.com/article_4800_31a82bd18b8f9305acb4ab756415bbc0.pdf