Applying systems theory to the evaluation of a whole school approach to violence prevention
Sarah Kearney A E , Loksee Leung A , Andrew Joyce B , Debbie Ollis C and Celia Green DA Our Watch, GPO Box 24229, Melbourne, Vic. 3001, Australia.
B Centre for Social Impact Swinburne, Swinburne University of Technology, H23, PO Box 218, Hawthorn, Vic. 3122, Australia.
C Deakin University, 221 Burwood Highway, Burwood, Vic. 3125, Australia.
D National Centre for Epidemiology and Population Health, Australian National University, Acton, ACT 2601, Australia.
E Corresponding author. Email: sarah.kearney@ourwatch.org.au
Health Promotion Journal of Australia 27(3) 230-235 https://doi.org/10.1071/HE16046
Submitted: 6 May 2016 Accepted: 19 July 2016 Published: 6 October 2016
Abstract
Issue addressed: Our Watch led a complex 12-month evaluation of a whole school approach to Respectful Relationships Education (RRE) implemented in 19 schools. RRE is an emerging field aimed at preventing gender-based violence. This paper will illustrate how from an implementation science perspective, the evaluation was a critical element in the change process at both a school and policy level.
Methods: Using several conceptual approaches from systems science, the evaluation sought to examine how the multiple systems layers – student, teacher, school, community and government – interacted and influenced each other. A distinguishing feature of the evaluation included ‘feedback loops’; that is, evaluation data was provided to participants as it became available. Evaluation tools included a combination of standardised surveys (with pre- and post-intervention data provided to schools via individualised reports), reflection tools, regular reflection interviews and summative focus groups.
Results: Data was shared during implementation with project staff, department staff and schools to support continuous improvement at these multiple systems levels. In complex settings, implementation can vary according to context; and the impact of evaluation processes, tools and findings differed across the schools. Interviews and focus groups conducted at the end of the project illustrated which of these methods were instrumental in motivating change and engaging stakeholders at both a school and departmental level and why.
Conclusion: The evaluation methods were a critical component of the pilot’s approach, helping to shape implementation through data feedback loops and reflective practice for ongoing, responsive and continuous improvement. Future health promotion research on complex interventions needs to examine how the evaluation itself is influencing implementation.
So what?: The pilot has demonstrated that the evaluation, including feedback loops to inform project activity, were an asset to implementation. This has implications for other health promotion activities, where evaluation tools could be utilised to enhance, rather than simply measure, an intervention. The findings are relevant to a range of health promotion research activities because they demonstrate the importance of meta-evaluation techniques that seek to understand how the evaluation itself was influencing implementation and outcomes.
Key words: evaluation methods, health promoting schools, interpersonal violence, participatory action research, program evaluation.
References
[1] Joyce A, Dabrowski A, Aston R, Carey G (2016) Evaluating for impact – what type of data can assist a health promoting school approach? Health Promot J Int| Evaluating for impact – what type of data can assist a health promoting school approach?Crossref | GoogleScholarGoogle Scholar |
[2] World Health Organization. Systems thinking for health systems strengthening. Geneva: Alliance for Health Policy and Systems Research; 2009.
[3] Meadows D. Leverage points: places to intervene in a system. Hartland, VT: Sustainability Institute; 1999.
[4] Mrazek P, Biglan A, Hawkins JD, Cody C. Community monitoring systems: tracking and improving the well-being of America’s children and adolescents. Fairfax: US Department of Health and Human Services; 2007.
[5] National Cancer Institute. Greater than the sum: systems thinking in tobacco control. National Cancer Institute. 2007. Available from: http://cancercontrol.cancer.gov/brp/tcrb/monographs/18/monograph18.html [Verified 4 August 2016].
[6] Green LW (2008) Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence? Fam Pract 25, i20–4.
| Making research relevant: if it is an evidence-based practice, where’s the practice-based evidence?Crossref | GoogleScholarGoogle Scholar | 18794201PubMed |
[7] Rowe AK (2009) Potential of integrated continuous surveys and quality management to support monitoring, evaluation, and the scale-up of health interventions in developing countries. Am J Trop Med Hyg 80, 971–9.
[8] Department of Education and Training (DET). Building respectful relationships: stepping out against gender-based violence. Melbourne: DET. 2014. Available from: http://www.education.vic.gov.au/school/teachers/health/Pages/respectrel.aspx [Verified 15 July 2016].
[9] Gleeson C, Kearney S, Leung L, Brislane J. Evidence paper: Respectful Relationships Education in schools. Melbourne: Our Watch. 2015. Available from: https://www.ourwatch.org.au/getmedia/4a61e08b-c958-40bc-8e02-30fde5f66a25/Evidence-paper-respectful-relationships-education-AA-updated.pdf.aspx [Verified 15 July 2016].
[10] Flood M, Fergus L, Heenan M. Respectful Relationships Education: violence prevention and Respectful Relationships Education in Victorian secondary schools. Melbourne: Victorian Department of Education and Early Childhood Development; 2009. Available from http://www.xyonline.net/sites/default/files/Flood,%20Respectful%20relationships%20education%2009.pdf [Verified 15 July 2016].
[11] Sundaram V, Maxwell C, Ollis D. Where does violence against women and girls work fit in? Exploring spaces for challenging violence within a sex-positive framework in schools. In V Sundaram, H Sauntson, editors. Global perspectives and key debates in sex and relationships education: addressing issues of gender, sexuality, plurality and power (pp. 68–83). London: Palgrave Macmillan; 2016.
[12] Maxwell C, Chase E, Warwick I, Aggleton P, Wharf H. Freedom to achieve: preventing violence, promoting equality: a whole school approach. London: Womankind Worldwide; 2010.
[13] Ollis D. Planning and delivering interventions to promote gender and sexuality. In Rivers I, Duncan N, editors. Bullying: experiences and discourses of sexuality and gender (pp. 145–161). London: Routledge; 2013.
[14] Ollis D, Harrison L (2016) Lessons in building capacity in sexuality education using the health promoting school framework. Health Educ 116, 138–53.
| Lessons in building capacity in sexuality education using the health promoting school framework.Crossref | GoogleScholarGoogle Scholar |
[15] St Leger L, Young I, Blanchard C, Perry M. Promoting health in school – from evidence to action. Paris: International Union for Health Promotion and Education (IUHPE); 2010. Available from: http://www.iuhpe.org/images/PUBLICATIONS/THEMATIC/HPS/Evidence-Action_ENG.pdf [Verified 15 July 2016].
[16] International Union for Health Promotion and Education (IUHPE). Achieving health promoting schools. Paris: IUHPE. 2009. Available from: http://www.iuhpe.org/images/PUBLICATIONS/THEMATIC/HPS/HPSGuidelines_ENG.pdf [Verified 15 July 2016].
[17] Our Watch, Australia’s National Research Organisation for Women’s Safety (ANROWS) and VicHealth. Change the story: a shared framework for the primary prevention of violence against women and their children in Australia. Melbourne: Our Watch; 2015. Available from: https://www.ourwatch.org.au/getmedia/1462998c-c32b-4772-ad02-cbf359e0d8e6/Change-the-story-framework-prevent-violence-women-children.pdf.aspx [Verified 15 July 2016]
[18] News ABC. Australian of the year: Rosie Batty awarded top honour for efforts to stop family violence. Sydney: ABC News. 2015. Available from: http://www.abc.net.au/news/2015-01-25/rosie-batty-named-australian-of-the-year-2015/6045290. [Verified 15 July 2016]
[19] Victorian Government. Royal commission into family violence. Melbourne: Victorian Government. 2016. Available from: http://www.rcfv.com.au/ [Verified 15 July 2016]
[20] Our Watch. Respectful Relationships Education in schools: the beginnings of change. 2016. Available from: https://www.ourwatch.org.au/What-We-Do/Respectful-relationships-education/RREiS-Evaluation-Report [Verified 15 August 2016].
[21] Boyatzis R. Transforming qualitative information. Thousand Oaks, CA: SAGE Publications; 1998.
[22] Neuendorf K. The content analysis guidebook. Thousand Oaks, CA: SAGE Publications; 2002.
[23] Ollis D, Harrison L, Richardson A. Building capacity in sexuality education: the Northern Bay College experience. Melbourne: School of Education, Deakin University; 2012.
[24] Bond L, Butler H. The Gatehouse Project: a multi-level integrated approach to promoting wellbeing in schools. In Killoran A, Kelly M, editors. Evidence-based public health: effectiveness and efficiency (pp. 250–69). Oxford: Oxford University Press; 2009.
[25] Senior L, Joyce A, Batras D. Becoming a health promoting school: using a ‘change agent’ to influence school structure, ethos, and ensure sustainability. In Simovska V, Mannix-McNamara P, editors. Schools for health and sustainability: theory, research and practice (pp. 131–53). Dordrecht, the Netherlands: Springer; 2014.
[26] Rogers PJ, Williams B. Evaluation for practice improvement and organizational learning. In Shaw IF, Greene JC, Melvin MM, editors. The SAGE Handbook of Evaluation (pp. 76–97). London: SAGE Publications; 2006.
[27] Adamowitsch M, Gugglberger L, Dur W (2014) Implementation practices in school health promotion: findings from an Austrian multiple-case study. Health Promot J Int
| Implementation practices in school health promotion: findings from an Austrian multiple-case study.Crossref | GoogleScholarGoogle Scholar |
[28] Kremser W (2011) Phases of school health promotion implementation through the lens of complexity theory: lessons learnt from an Austrian case study. Health Promot J Int 26, 136–47.
| Phases of school health promotion implementation through the lens of complexity theory: lessons learnt from an Austrian case study.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BC3MvpsVaqug%3D%3D&md5=f4d276280d4f0d097a402816391b06daCAS |
[29] Belansky ES, Cutforth N, Chavez R, Crane LA, Waters E, Marshall JA (2013) Adapted intervention mapping: a strategic planning process for increasing physical activity and healthy eating opportunities in schools via environment and policy change. J Sch Health 83, 194–205.
| Adapted intervention mapping: a strategic planning process for increasing physical activity and healthy eating opportunities in schools via environment and policy change.Crossref | GoogleScholarGoogle Scholar | 23343320PubMed |
[30] Dooris M, Poland B, Kolbe L, De Leeuw E, McCall D, Wharf-Higgins J. Global perspectives on health promotion effectiveness. New York: Springer Science and Business Media; 2007.