Building research and evaluation capacity in population health: the NSW Health approach
Barry Edwards A D , Beth Stickney A , Andrew Milat A B , Danielle Campbell A and Sarah Thackway A CA Centre for Epidemiology and Evidence, New South Wales Ministry of Health, 73 Miller Street, North Sydney, NSW 2060, Australia.
B Sydney Medical School, Edward Ford Building A27, University of Sydney, NSW 2006, Australia.
C School of Public Health and Community Medicine, University of New South Wales, UNSW Sydney, NSW 2052, Australia.
D Corresponding author. Email: baedw@doh.health.nsw.gov.au
Health Promotion Journal of Australia 27(3) 264-267 https://doi.org/10.1071/HE16045
Submitted: 6 May 2016 Accepted: 26 August 2016 Published: 19 October 2016
Abstract
Issue addressed: An organisational culture that values and uses research and evaluation (R&E) evidence to inform policy and practice is fundamental to improving health outcomes. The 2016 NSW Government Program Evaluation Guidelines recommend investment in training and development to improve evaluation capacity. The purpose of this paper is to outline the approaches taken by the NSW Ministry of Health to develop R&E capacity and assess these against existing models of practice.
Method: The Ministry of Health’s Centre for Epidemiology and Evidence (CEE) takes an evidence-based approach to building R&E capacity in population health. Strategies are informed by: the NSW Population Health Research Strategy, R&E communities of practice across the Ministry and health Pillar agencies and a review of the published evidence on evaluation capacity building (ECB). An internal survey is conducted biennially to monitor research activity within the Ministry’s Population and Public Health Division. One representative from each of the six centres that make up the Division coordinates completion of the survey by relevant staff members for their centre.
Results: The review identified several ECB success factors including: implementing a tailored multifaceted approach; an organisational commitment to R&E; and offering experiential training and ongoing technical support to the workforce. The survey of research activity found that the Division funded a mix of research assets, research funding schemes, research centres and commissioned R&E projects. CEE provides technical advice and support services for staff involved in R&E and in 2015, 22 program evaluations were supported. R&E capacity building also includes a series of guides to assist policy makers, practitioners and researchers to commission, undertake and use policy-relevant R&E. Staff training includes workshops on critical appraisal, program logic and evaluation methods. From January 2013 to June 2014 divisional staff published 84 peer-reviewed papers and one book chapter.
Conclusion: A strategic approach to R&E capacity building compares favourably with organisational dimensions of ECB and has facilitated the generation of high quality population health R&E in NSW.
So what?: An evidence-based multistrategy approach to population health R&E can result in substantial contributions to the population-health evidence base.
Key words: capacity building, evaluation methods, evidenced-based practice, health policy, program evaluation.
References
[1] Oxman AD, Bjørndal A, Becerra-Posada F, Gibson M, Block MA, Haines A, et al (2010) A framework for mandatory impact evaluation to ensure well informed public policy decisions. Lancet 375, 427–31.| A framework for mandatory impact evaluation to ensure well informed public policy decisions.Crossref | GoogleScholarGoogle Scholar | 20113827PubMed |
[2] NSW Department of Premier and Cabinet. NSW Government Program Evaluation Guidelines: January 2016. Sydney: NSW Department of Premier and Cabinet; 2016.
[3] NSW Department of Premier and Cabinet. Circular C2016–01: Program Evaluation. Sydney: NSW Department of Premier and Cabinet; 2016.
[4] Chalmers I (2003) Trying to do more good than harm in policy and practice: the role of rigorous, transparent, up-to-date evaluations. Ann Am Acad Pol Soc Sci 589, 22–40.
| Trying to do more good than harm in policy and practice: the role of rigorous, transparent, up-to-date evaluations.Crossref | GoogleScholarGoogle Scholar |
[5] Productivity Commission. Strengthening evidence-based policy in the Australian Federation, Volume 2: Background Paper. Canberra: Commonwealth of Australia; 2010.
[6] Economics Division, Queensland Treasury and Trade. Queensland Government Program Evaluation Guidelines November 2014. Brisbane: Queensland Government; 2014.
[7] Labin SL, Duffy JL, Meyers DC, Wandersman A, Lesesne CA (2012) A research synthesis of the evaluation capacity building literature. Am J Eval 33, 307–38.
| A research synthesis of the evaluation capacity building literature.Crossref | GoogleScholarGoogle Scholar |
[8] Naccarella L, Pirkis J, Kohn F, Morley B, Burgess P, Blashki G (2007) Building evaluation capacity: definitional and practical implications from an Australian case study. Eval Program Plann 30, 231–6.
| Building evaluation capacity: definitional and practical implications from an Australian case study.Crossref | GoogleScholarGoogle Scholar | 17689328PubMed |
[9] Adams J, Dickinson P (2010) Evaluation training to build capability in the community and public health workforce. Am J Eval 31, 421–33.
| Evaluation training to build capability in the community and public health workforce.Crossref | GoogleScholarGoogle Scholar |
[10] Research and Evaluation Unit. Building Evaluation Capacity in DIAC 2010–2013: a strategy to build our capacity to conduct effective evaluations to support program management and inform decision making. Canberra: Department of Immigration and Citizenship; 2010.
[11] Bamberg J, Perlesz A, McKenzie P, Read S (2010) Utilising implementation science in building research and evaluation capacity in community health. Aust J Prim Health 16, 276–83.
| Utilising implementation science in building research and evaluation capacity in community health.Crossref | GoogleScholarGoogle Scholar | 21138694PubMed |
[12] Levine R, Russ-Eft D, Burling A, Stephens J, Downey J (2013) Evaluating health services research capacity building programs: implications for health services and human resource development. Eval Program Plann 37, 1–11.
| Evaluating health services research capacity building programs: implications for health services and human resource development.Crossref | GoogleScholarGoogle Scholar | 23291786PubMed |
[13] Norton S, Milat A, Edwards B, Giffin M (2016) Narrative review of strategies by organizations for building evaluation capacity. Eval Program Plann 58, 1–19.
| Narrative review of strategies by organizations for building evaluation capacity.Crossref | GoogleScholarGoogle Scholar | 27258905PubMed |
[14] Volkov B, King J. A checklist for building organizational evaluation capacity. Western Michigan University; 2007. Available from: https://www.wmich.edu/sites/default/files/attachments/u350/2014/organiziationevalcapacity.pdf [Verified 27 July 2016].
[15] Taylor-Ritzler T, Suarez-Balcazar Y, Garcia-Iriarte E, Henry D, Balcazar F (2013) Understanding and measuring evaluation capacity: a model and instrument validation study. Am J Eval 34, 190–206.
| Understanding and measuring evaluation capacity: a model and instrument validation study.Crossref | GoogleScholarGoogle Scholar |
[16] NSW Department of Health. Promoting the generation and effective use of population health research in NSW: A strategy for NSW Health 2011–2015. Sydney: NSW Department of Health; 2010.
[17] Centre for Epidemiology and Evidence. NSW Ministry of Health and Pillars – evaluation capability building plan. Sydney: NSW Ministry of Health; 2016.
[18] Fourney A, Gregson J, Sugerman S, Bellow A (2011) Building evaluation capacity in local programs for multisite nutrition education interventions. J Nutr Educ Behav 43, S130–6.
| Building evaluation capacity in local programs for multisite nutrition education interventions.Crossref | GoogleScholarGoogle Scholar | 21683282PubMed |
[19] Centre for Epidemiology and Evidence. Commissioning Evaluation Services: A Guide. Sydney: NSW Ministry of Health; 2015. Available from: http://www.health.nsw.gov.au/research/Pages/evaluation-guide.aspx [Verified 5 May 2016].
[20] Centre for Epidemiology and Evidence. Commissioning Editorial Services: A Guide. Sydney: NSW Ministry of Health; 2012. Available from: http://www.health.nsw.gov.au/research/Publications/commissioning-editorial-services.pdf [Verified 5 May 2016].
[21] Centre for Epidemiology and Evidence. Increasing the scale of population health interventions: A Guide. Sydney: NSW Ministry of Health; 2014. Available from: http://www.health.nsw.gov.au/research/Publications/scalability-guide.pdf [Verified 5 May 2016].