Evolution of a multilevel framework for health program evaluation
Malcolm Masso A B , Karen Quinsey A and Dave Fildes A
+ Author Affiliations
- Author Affiliations
A Centre for Health Service Development, Australian Health Services Research Institute, Building 234 (iC Enterprise 1), Innovation Campus, University of Wollongong, Wollongong, NSW 2522, Australia. Email: kquinsey@uow.edu.au; dfildes@uow.edu.au
B Corresponding author. Email: mmasso@uow.edu.au
Australian Health Review 41(3) 239-245 https://doi.org/10.1071/AH15117
Submitted: 18 June 2015 Accepted: 10 May 2016 Published: 16 June 2016
Abstract
A well-conceived evaluation framework increases understanding of a program’s goals and objectives, facilitates the identification of outcomes and can be used as a planning tool during program development. Herein we describe the origins and development of an evaluation framework that recognises that implementation is influenced by the setting in which it takes place, the individuals involved and the processes by which implementation is accomplished. The framework includes an evaluation hierarchy that focuses on outcomes for consumers, providers and the care delivery system, and is structured according to six domains: program delivery, impact, sustainability, capacity building, generalisability and dissemination. These components of the evaluation framework fit into a matrix structure, and cells within the matrix are supported by relevant evaluation tools. The development of the framework has been influenced by feedback from various stakeholders, existing knowledge of the evaluators and the literature on health promotion and implementation science. Over the years, the framework has matured and is generic enough to be useful in a wide variety of circumstances, yet specific enough to focus data collection, data analysis and the presentation of findings.
What is known about the topic? Evaluation in healthcare typically investigates the implementation of complex innovations in uncontrolled ‘real world’ settings. This presents many challenges for evaluators and those wanting to commission evaluations.
What does this paper add? Herein we describe the origins and development of an evaluation framework that is structured in terms of three levels and six domains. The framework has stood the test of time and been used to evaluate a variety of innovations in the delivery of health and aged care. Novel aspects of the framework include the concepts of levels, capacity building and sustainability.
What are the implications for practitioners? The structure of the evaluation framework is simple, yet comprehensive, and is suitable for use in a wide range of settings. The framework can also be used to both evaluate and plan the implementation of innovations in health service delivery.
Additional keywords: capacity building, evaluation framework, impact, implementation, sustainability.
References
[1] Bledsoe KL, Graham JA. The use of multiple evaluation approaches in program evaluation. Am J Eval 2005; 26 302–19.| The use of multiple evaluation approaches in program evaluation.Crossref | GoogleScholarGoogle Scholar |
[2] Preskill H, Boyle S. A multidisciplinary model of evaluation capacity building. Am J Eval 2008; 29 443–59.
| A multidisciplinary model of evaluation capacity building.Crossref | GoogleScholarGoogle Scholar |
[3] Patton MQ. A context and boundaries for a theory-driven approach to validity. Eval Program Plann 1989; 12 375–7.
| A context and boundaries for a theory-driven approach to validity.Crossref | GoogleScholarGoogle Scholar |
[4] Perkins D, Owen A, Cromwell D, Adamson L, Eagar K, Quinsey K, Green J. The lllawarra Coordinated Care Trial: better outcomes with existing resources? Aust Health Rev 2001; 24 172–78.
| The lllawarra Coordinated Care Trial: better outcomes with existing resources?Crossref | GoogleScholarGoogle Scholar |
[5] Pettigrew A, Ferlie E, McKee L. Shaping strategic change. London: SAGE Publications; 1992.
[6] Eagar K, Owen A, Perkins D, Adamson L, Quinsey K, Harvey R, Green J., et al. The Care Net Trial – the evaluators conclusions. Report 10 of the final evaluation of the Care Net Illawarra Coordinated Care Trial. Wollongong: Centre for Health Service Development, University of Wollongong; 2000.
[7] Eagar K, Owen A, Perkins D, Burgess P, Epstein M, Adamson L, Quinsey K. Planning guidelines for national demonstration projects in integrated mental health care. Canberra: Commonwealth of Australia; 1999.
[8] Owen A, Perkins D, Senior K, Eagar K. The Griffith Area Palliative Care Service: a baseline assessment of its evaluability, sustainability and generalisability. Wollongong: Centre for Health Service Development, University of Wollongong; 2001.
[9] Cromwell D, Senior K, Owen A, Gordon R, Eagar K. Can the National Palliative Care Strategy be translated into a model of care that works for rural Australia? An answer from the Griffith Area Palliative Care Service (GAPS) experience. Wollongong: Centre for Health Service Development, University of Wollongong; 2003.
[10] Hawe P, King L, Noort M, Jordens C, Lloyd B. Indicators to help with capacity building in health promotion. Sydney: NSW Health Department; 2000.
[11] Hawe P, King L, Noort M, Gifford SM, Lloyd B. Working invisibly: health workers talk about capacity-building in health promotion. Health Promot Int 1998; 13 285–95.
| Working invisibly: health workers talk about capacity-building in health promotion.Crossref | GoogleScholarGoogle Scholar |
[12] Hawe P, Noort M, King L, Jordens C. Multiplying health gains: the critical role of capacity-building within health promotion programs. Health Policy 1997; 39 29–42.
| Multiplying health gains: the critical role of capacity-building within health promotion programs.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DyaK2s7ptFykug%3D%3D&md5=38fd02f07ff41997e8f5ea792cae4504CAS | 10164903PubMed |
[13] Quinsey K, Williams K, Fildes D, Masso M, Senior K, Yeatman H, Eagar K. Caring communities: evaluation of a national palliative care program. Wollongong: Centre for Health Service Development, University of Wollongong; 2006.
[14] Quinsey K, Masso M, Fildes D, Siminski P, Grootemaat P, Matete S, Eagar K. Providing palliative care in rural Australia: results of a national program evaluation. Wollongong: Centre for Health Service Development, University of Wollongong; 2007.
[15] Eagar K, Cranny C, Fildes D. Evaluation and palliative care: a guide to the evaluation of palliative care services and programs. Wollongong: Centre for Health Service Development, University of Wollongong; 2004.
[16] Eagar K, Senior K, Fildes D, Quinsey K, Owen A, Yeatman H, Gordon R, Posner N. The Palliative Care Evaluation Tool Kit: a compendium of tools to aid in the evaluation of palliative care projects. Wollongong: Centre for Health Service Development, University of Wollongong; 2003.
[17] Quinsey K, Williams K, Westera A, Morris D. Care planning sub-program: findings from the national evaluation. Wollongong: Centre for Health Service Development, University of Wollongong; 2009.
[18] Masso M, Westera A, Quinsey K, Morris D, Pearse J. Encouraging best practice in residential aged care program: final evaluation report. Wollongong: Centre for Health Service Development, University of Wollongong; 2011.
[19] Yeatman H, Quinsey K, Dawber J, Nielsen W, Condon-Paoloni D, Eckermann S, Morris D, Grootemaat P, Fildes D. Combining realism with rigour: an evaluation of a national kitchen garden program in Australian primary schools. Eval J Australas 2014; 14 17–24.
[20] Westera A, Stevermuer T, Samsa P, Quinsey K, Owen A, Marosszeky N, Eagar K. The NSW SAFTE Care Program: evaluation of a pilot program to prevent unnecessary hospital attendances by older people. Wollongong: Centre for Health Service Development, University of Wollongong; 2007.
[21] Eagar K, Green J, Quinsey K, Masso M, Owen A, Grootemaat P, Thompson C. Evaluation of the Gold Coast Hospital Avoidance Program: final report. Wollongong: Centre for Health Service Development, University of Wollongong; 2009.
[22] Thompson C, Quinsey K, Gordon R, Williams K, Eckermann S, Andersen P, Snoek M, Eagar K. Health Workforce Australia Expanded Scopes of Practice Program: evaluation framework. Wollongong: Centre for Health Service Development, University of Wollongong; 2012.
[23] Thompson C, Williams K, Masso M. HWA Expanded Scopes of Practice Program Evaluation: national synthesis. Wollongong: Centre for Health Service Development, University of Wollongong; 2014.
[24] Ferlie E, Shortell SM. Improving the quality of health care in the United Kingdom and the United States: a framework for change. Milbank Q 2001; 79 281–315.
| Improving the quality of health care in the United Kingdom and the United States: a framework for change.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BD3Mzos1ajsQ%3D%3D&md5=3762f9d0176ad88534cd082dcb289a0eCAS | 11439467PubMed |
[25] Grol R, Wensing M. What drives change? Barriers to and incentives for achieving evidence-based practice. Med J Aust 2004; 180 S57–60.
| 15012583PubMed |
[26] Damschroder L, Aron D, Keith R, Kirsh S, Alexander J, Lowery J. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4 50
| Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.Crossref | GoogleScholarGoogle Scholar | 19664226PubMed |
[27] Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004; 82 581–629.
| Diffusion of innovations in service organizations: systematic review and recommendations.Crossref | GoogleScholarGoogle Scholar | 15595944PubMed |
[28] Swerissen H, Crisp BR. The sustainability of health promotion interventions for different levels of social organization. Health Promot Int 2004; 19 123–30.
| The sustainability of health promotion interventions for different levels of social organization.Crossref | GoogleScholarGoogle Scholar | 14976180PubMed |
[29] Eagar K, Owen A, Masso M, Quinsey K. The Griffith Area Palliative Care Service (GAPS): an evaluation of an Australian rural palliative care model. Prog Palliat Care 2006; 14 112–19.
| The Griffith Area Palliative Care Service (GAPS): an evaluation of an Australian rural palliative care model.Crossref | GoogleScholarGoogle Scholar |
[30] Masso M, Fildes D, Quinsey K, Matete S. GAPS revisited: follow up evaluation of an Australian rural palliative care service. Prog Palliat Care 2007; 15 233–9.
| GAPS revisited: follow up evaluation of an Australian rural palliative care service.Crossref | GoogleScholarGoogle Scholar |
[31] Thompson C, Williams K, Morris D, Lago L, Kobel C, Quinsey K, Eckermann S, Andersen P, Masso M. HWA Expanded Scopes of Practice Program Evaluation: extending the role of paramedics sub-project final report. Wollongong: Centre for Health Service Development, Australian Health Services Research Institute, University of Wollongong; 2014.
[32] Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F. Implementation research: a synthesis of the literature. FMHI Publication #231. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; 2005.
[33] Cochrane Effective Practice and Organisation of Care Review Group (EPOC). Data collection checklist. Ottawa: Cochrane EPOC; 2007.
[34] Elkhuizen SG, Limburg M, Bakker PJM, Klazinga NS. Evidence-based re-engineering: re-engineering the evidence. A systematic review of the literature on business process redesign (BPR) in hospital care. Int J Health Care Qual Assur 2006; 19 477–99.
| Evidence-based re-engineering: re-engineering the evidence. A systematic review of the literature on business process redesign (BPR) in hospital care.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BD28nmsFelsQ%3D%3D&md5=a59386629081c5320c881ae576b81b56CAS |
[35] Masso M, Marcolin S, Carolan J, Quinsey K, Lago L, Robert R, Bate P, Eagar K. Third annual report on the NSW Clinical Services Redesign Program. Wollongong: Centre for Health Service Development, University of Wollongong; 2008.
[36] Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci 2012; 7 17
[37] Buchanan D, Fitzgerald L, Ketley D, Gollop R, Jones JL, Lamont SS, Neath A, Whitby E., et al No going back: a review of the literature on sustaining organizational change. Int J Manag Rev 2005; 7 189–205.
| No going back: a review of the literature on sustaining organizational change.Crossref | GoogleScholarGoogle Scholar |
[38] Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res 1998; 13 87–108.
| Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DyaK1c3jvVKitA%3D%3D&md5=0484b567ed75e7ef5047519edc662e31CAS | 10178339PubMed |
[39] Scheirer MA. Is sustainability possible? A review and commentary on empirical studies of program sustainability. Am J Eval 2005; 26 320–47.
| Is sustainability possible? A review and commentary on empirical studies of program sustainability.Crossref | GoogleScholarGoogle Scholar |
[40] Maher L, Gustafson D, Evans A. NHS sustainability model. London: NHS Institute for Innovation and Improvement; 2010.
[41] Meyer A-M, Davis M, Mays GP. Defining organizational capacity for public health services and systems research. J Public Health Manag Pract 2012; 18 535–44.
| Defining organizational capacity for public health services and systems research.Crossref | GoogleScholarGoogle Scholar | 23023278PubMed |
[42] LaFond AK, Brown L, Macintyre K. Mapping capacity in the health sector: a conceptual framework. Int J Health Plann Manage 2002; 17 3–22.
| Mapping capacity in the health sector: a conceptual framework.Crossref | GoogleScholarGoogle Scholar | 11963442PubMed |
[43] Cassidy EF, Leviton LC, Hunter DEK. The relationships of program and organizational capacity to program sustainability: what helps programs survive? Eval Program Plann 2006; 29 149–52.
| The relationships of program and organizational capacity to program sustainability: what helps programs survive?Crossref | GoogleScholarGoogle Scholar |
[44] Schuh RG, Leviton LC. A framework to assess the development and capacity of non-profit agencies. Eval Program Plann 2006; 29 171–9.
| A framework to assess the development and capacity of non-profit agencies.Crossref | GoogleScholarGoogle Scholar |
