What is the value of hospital mortality indicators, and are there ways to do better?
Anna Barker A , Kerrie Mengersen B and Anthony Morton C DA Centre of Research Excellence in Patient Safety, Department of Epidemiology and Preventive Medicine, Monash University, The Alfred Centre, 99 Commercial Rd, Melbourne, VIC. 3004, Australia.
B Faculty of Science and Technology, Queensland University of Technology, George St, Brisbane, QLD 4000, Australia.
C Infection Management Services, Princess Alexandra Hospital, Ipswich Rd, Woolloongabba, QLD 4102, Australia.
D Corresponding author. Email: amor5444@bigpond.net.au
Australian Health Review 36(4) 374-377 https://doi.org/10.1071/AH11132
Submitted: 23 December 2011 Accepted: 13 May 2012 Published: 2 November 2012
Abstract
Monitoring hospital performance using patient safety indicators is one of the key components of healthcare reform in Australia. Mortality indicators, including the hospital standardised mortality ratio and deaths in low mortality diagnosis reference groups have been included in the core national hospital-based outcome indicator set recommended for local generation and review and public reporting. Although the face validity of mortality indicators such as these is high, an increasing number of studies have demonstrated that there are concerns regarding their internal, construct and criterion validity. Use of indicators with poor validity has the consequence of potentially incorrectly classifying hospitals as performance outliers and expenditure of limited hospital staff time on activities which may provide no gain to hospital quality and safety and may in fact cause damage to morale. This paper reviews the limitations of current approaches to monitoring hospital quality and safety performance using mortality indicators. It is argued that there are better approaches to improving performance than monitoring with mortality indicators generated from hospital administrative data. These approaches include use of epidemiologically sound, clinically relevant data from clinical-quality registries, better systems of audit, evidence-based bundles, checklists, simulators and application of the science of complex systems.
What is known about the topic? Public reporting of adverse events such as hospital standardised mortality ratios deaths in low mortality diagnosis reference groups is a key component of Australian healthcare reform. There is much debate in Australia and internationally concerning the appropriateness of this approach.
What does the paper add? We extend the current literature and debate by reviewing the statistical limitations, challenges and biases inherent in these indicators. Alternatives for quality and safety performance monitoring that are more robust are presented.
What are the implications for practitioners? The hospital standardised mortality ratio and death in low mortality diagnosis reference groups indicators should be used with extreme caution. Although public reporting of quality and safety indicators is necessary there are likely to be better methods to detect substandard performance. These include: properly structured morbidity and mortality meetings, independent audits, evidence-based bundles and checklists, sequential data analysis (e.g. using CUSUMS), and the use of simulators. To achieve maximum safety it is necessary, in addition to using these methods, to understand the characteristics of hospitals as complex systems that exhibit safe emergent behaviour, e.g. using the science of complex systems and its tools. Genuine safety cannot be achieved simply be studying ‘unsafety’. In addition, epidemiologically sound, clinically relevant clinical-quality registries are required.
References
[1] Barker A, Brand C, Evans S, Cameron P, Jolley D. Death in low-mortality diagnosis-related groups: frequency, and the impact of patient and hospital characteristics. Med J Aust 2011; 195 89–94.[2] Black N. Assessing the quality of hospitals. Hospital standardised mortality ratios should be abandoned. BMJ 2010; 340 933–4.
[3] Lilford R, Pronovost P. Using hospital mortality rates to judge hospital performance: a bad idea that just won’t go away. BMJ 2010; 340 c2016
| Using hospital mortality rates to judge hospital performance: a bad idea that just won’t go away.Crossref | GoogleScholarGoogle Scholar |
[4] Bottle A, Jarman B, Aylin P. Strengths and weaknesses of hospital standardised mortality ratios. BMJ 2011; 342 749–53.
[5] Gallagher M, Krumholz H. Public reporting of hospital outcomes: a challenging road ahead. Med J Aust 2011; 194 658–60.
[6] Ben-Tovim D. Asking the hard questions about safety and quality indicators. Med J Aust 2011; 194 623–4.
[7] Scott I, Brand C, Phelps G, Barker A, Cameron P. Using hospital standardised mortality ratios to assess quality of care – proceed with extreme caution. Med J Aust 2011; 194 645–8.
[8] Mihrshahi S, Brand C, Ibrahim J, Evans S, Jolley D, Cameron P. Validity of the indicator ‘death in low-mortality diagnosis-related groups’ for measuring patient safety and healthcare quality in hospitals. Intern Med J 2010; 40 250–7.
| Validity of the indicator ‘death in low-mortality diagnosis-related groups’ for measuring patient safety and healthcare quality in hospitals.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BC3cvosV2gtA%3D%3D&md5=4903e7c80fc40a988db0a440ef9d2f62CAS |
[9] Evans S, Scott I, Johnson N, Cameron P, McNeil J. Development of clinical-quality registries in Australia: the way forward. Med J Aust 2011; 194 360–3.
[10] Morton A, Cook D, Mengersen K, Waterhouse M. Limiting risk of hospital adverse events: avoiding train wrecks is more important than counting and reporting them. J Hosp Infect 2010; 76 283–6.
| Limiting risk of hospital adverse events: avoiding train wrecks is more important than counting and reporting them.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BC3cbntVeqtA%3D%3D&md5=8860d384efb699a420177d750995876fCAS |
[11] Morton A, Mengersen K, Waterhouse M, Steiner S, Looke D. Sequential analysis of uncommon adverse outcomes. J Hosp Infect 2010; 76 114–8.
| Sequential analysis of uncommon adverse outcomes.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BC3cfgtFSnsQ%3D%3D&md5=4391bad06f9b2b1de68fbfc0b054f43dCAS |
[12] Dunbar J, Reddy P, May S. Deadly healthcare. Bowen Hills: Australian Academic Press, 2011.
[13] Singer A. Mandatory regular meetings of hospital staff would complement medical audit and revalidation. BMJ 2000; 320 1072.1
| Mandatory regular meetings of hospital staff would complement medical audit and revalidation.Crossref | GoogleScholarGoogle Scholar |
[14] Vincent C. Patient safety. 2nd edn. Chichester: Wiley–Blackwell/BMJ Books, 2010.
[15] Thompson A, Stonebridge L. Building a framework for trust: critical event analysis for deaths in surgical care. BMJ 2005; 330 1139–42.
| Building a framework for trust: critical event analysis for deaths in surgical care.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BD2M3ltFSmug%3D%3D&md5=be44be7dd9bab5b935c9db8af498e71aCAS |
[16] Gawande A. The checklist manifesto. London: Profile Books, 2010.
[17] Foot C. Australian Anaesthesia 2007; www.anzca.edu.au/resources/college-publications/pdfs/books-and-publications/Australasian%20Anaesthesia/australasian-anaesthesia-2007/Foot.pdf
[18] Australian Academy of Science and CSIRO. Agent based modelling. Available at: www.csiro.au/science/CABM [Verified 17 September 2012.]
[19] Healy J, Dugdale P. Patient safety first. p. 18. Crows Nest: Allen and Unwin, 2009.
[20] Bland M, Altman G. Statistics notes: some examples of regression towards the mean. BMJ 1994; 309 780
| Statistics notes: some examples of regression towards the mean.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DyaK2M%2Fjsl2mug%3D%3D&md5=64ab1b42f028504c7c6becc08001d336CAS |
[21] Barker A, Brand C, Haines T, Hill K, Brauer S, Jolley D, et al The 6-PACK programme to decrease fall-related injuries in acute hospitals: protocol for a cluster randomised controlled trial. Inj Prev 2011; 17 e5
| The 6-PACK programme to decrease fall-related injuries in acute hospitals: protocol for a cluster randomised controlled trial.Crossref | GoogleScholarGoogle Scholar |
[22] Institute of Healthcare Improvement. The improvement map. http://app.ihi.org/imap/tool [Verified 17 September 2012]
[23] Gilbert N. Agent-based models. Los Angeles: SAGE Publications, 2008.
[24] Temime L, Opatowski L, Pannet Y, Brun-Buisson C, Boëlle P, Guillemot D. Peripatetic health-care workers as potential superspreaders. Proc Natl Acad Sci USA 2009; 106 18 420–5.
| Peripatetic health-care workers as potential superspreaders.Crossref | GoogleScholarGoogle Scholar | 1:CAS:528:DC%2BD1MXhsVaiurnN&md5=443fe25fd250f55e3bdb1ea16c2cc7b8CAS |
[25] Galea S, Riddle M, Kaplan G. Causal thinking and complex system approaches in epidemiology. Int J Epidemiol 2010; 39 97–106.
| Causal thinking and complex system approaches in epidemiology.Crossref | GoogleScholarGoogle Scholar |
[26] Spiegelhalter D. Handling overdispersion of performance indicators. Qual Saf Health Care 2005; 14 347–51.
| Handling overdispersion of performance indicators.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BD2Mrit1Wiuw%3D%3D&md5=38857652887eabfcc73dd438c394e099CAS |
[27] Jones H, Ohlssen D, Spiegelhalter D. Use of false discovery rate when comparing multiple health care providers. J Clin Epidemiol 2008; 61 232–40.
| Use of false discovery rate when comparing multiple health care providers.Crossref | GoogleScholarGoogle Scholar |
[28] Winkel P, Zhang N. Statistical development of quality in medicine. Chichester: John Wiley and Sons, 2007.
[29] Morton A, Mengersen K, Waterhouse M, Steiner S. Analysis of aggregated hospital infection data for accountability. J Hosp Infect 2010; 76 287–91.
| Analysis of aggregated hospital infection data for accountability.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BC3cbntVygsQ%3D%3D&md5=fc136e5c2bd0db013362c7d7f26e7b35CAS |
[30] Adab P, Rouse A, Mohammed M, Marshall T. Performance league tables: the NHS deserves better. BMJ 2002; 324 95–8.
| Performance league tables: the NHS deserves better.Crossref | GoogleScholarGoogle Scholar |
[31] Shahian D, Normand S. Comparison of risk-adjusted hospital outcomes. Circulation 2008; 117 1955–63.
| Comparison of risk-adjusted hospital outcomes.Crossref | GoogleScholarGoogle Scholar |
[32] Nelson L. Notes on the Shewhart control chart. Journal of Quality Technology 1999; 31 124–6.
[33] Mitchell M. Complexity. Chapter 15. New York: Oxford University Press 2009.
[34] Silcocks P. Estimating confidence limits on a standardised mortality ratio when the expected number is not error free. J Epidemiol Community Health 1994; 48 313–7.
| Estimating confidence limits on a standardised mortality ratio when the expected number is not error free.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DyaK2czivVSrug%3D%3D&md5=05cbb21e57b5e4568fb562141b0cc25dCAS |
[35] Faris P, Ghali W, Brant R. Bias in estimates of confidence intervals for health outcome report cards. J Clin Epidemiol 2003; 56 553–8.
| Bias in estimates of confidence intervals for health outcome report cards.Crossref | GoogleScholarGoogle Scholar |
[36] Ng J, Morlet N, Bremmer A, Bulsara M, Morton A, Semmens J. Techniques to monitor endophthalmitis and other cataract surgery complications. Ophthalmology 2008; 115 3–10.
| Techniques to monitor endophthalmitis and other cataract surgery complications.Crossref | GoogleScholarGoogle Scholar |
