Advancing evaluation practice in health promotion

Program evaluation has long been recognised as a core competency for health promotion practitioners,^1,2 reflecting the vital contribution that this evaluation can make to the design, impact and sustainability of our policies and strategies. Over the past three decades, several popular textbooks^3,4 and a range of guides and frameworks^5–7 have been produced to facilitate appropriate and high-quality evaluation in this field. Given this history, which spans much of the contemporary health promotion movement, it is of interest to see the renewed and critical attention to evaluation practice, methods, use and capacity by health promotion agencies and practitioners. The questions of what is generating this interest, what has been revealed by recent reviews and analysis, and what further learning and resources may strengthen evaluation, warrant consideration in this special issue of the Health Promotion Journal of Australia on ‘Advancing evaluation practice’.

Some commentators have argued that there are few examples of comprehensive evaluation in health promotion in Australia and elsewhere, or of evaluation evidence being used to guide program development.⁸ This represents a lost opportunity for learning and program improvement. Others have argued that the cuts to funding of health promotion agencies and programs in Australia in recent years demonstrate the need for higher quality evidence of the public health contribution of health promotion policies and strategies.⁹ A further perspective has highlighted the complex policy and practice challenges that face health promotion agencies, and the need for closer attention to how evaluation capacity can be developed to provide the evidence needed to guide this work.¹⁰

Recent reviews of published and unpublished evaluation reports from Australian health promotion projects have shed light on the scope and methods that characterise evaluation practice. Hulme-Chambers et al.⁹ identified 157 articles in peer-reviewed journals from 1992–2011 that reported on health promotion evaluations. Impact evaluation was most often presented, with about half of the evaluations using one method only (usually surveys). Notably, there was little change in the purposes and designs of published evaluations over this 20-year period. Francis and Smith¹¹ audited unpublished evaluation reports provided by 24 health promotion organisations in Melbourne between 2008 and 2011. They found that all reported process evaluation, most included impact evaluation, and formative evaluation was rare. There was limited detail given in the reports about impact evaluation methods, but where described, this most often entailed pre- and post-surveys in small samples, participant interviews or focus groups.

Studies that have explored factors affecting the quality and extent of evaluation by health promotion agencies have confirmed that time, resources, staff skills, manager priorities and the presence of an evaluation culture are commonly reported issues.^12,13 More recently, interviews with senior policymakers and evaluators in Australia have highlighted the role played by political imperatives in determining whether evaluation is feasible, given the pressure to be demonstrating action within short timeframes, or even desirable, because of the risk that it may reveal that an initiative has had limited success.¹⁰ Another study involving interviews with practitioners found that the narrow reporting requirements set by funding agencies, and changes to reporting priorities, were deterrents to comprehensive evaluation.¹¹ This study also found that difficulties in defining and measuring impacts affected the ability of agencies to evaluate projects to the extent that they would like.

Against this background, this special issue has been prepared to disseminate recent insights that can assist the planning, design and implementation of evaluations of health promotion policies and strategies. The first section of the issue presents three papers that explore evaluation designs and frameworks, with two of these focusing on strategies delivered via online and mobile technologies. In their systematic review of the methods used to evaluate health promotion via social networking sites, Lim et al.¹⁴ report that evidence and practice insights can be gained through rigorous testing of these in real-life settings, using quasi-experimental or before–after designs together with comprehensive engagement metrics. White et al.¹⁵ describe four models that can be used to evaluate mHealth interventions (using mobile technologies), and present a case study of the plan developed to evaluate the Milk Man app, an innovative approach to engaging fathers in the promotion of breastfeeding. This illustrates how app design principles, technological performance principles, and behavioural and health outcomes can all be examined within the scope of an evaluation. In the third paper, Wolfenden et al.¹⁶ argue that the speed of evidence generation to inform policy and practice can be improved through greater understanding and use of effectiveness–implementation hybrid designs. Three types of hybrid design are presented, with examples of how they may be used in health promotion evaluation.

Addressing the challenges of conceptualising and reporting on the implementation and impacts of health promotion strategies, four papers in this issue present novel methods for data collection. In the first of these, Kostadinov et al.¹⁷ report the use of a perceived community leadership readiness tool to understand implementation quality and context across 20 communities in the South Australia Obesity Prevention and Lifestyle program. Reilly et al.¹⁸ examine the properties of four measures for assessing the characteristics of school canteen menus, and show that the quick menu audit tool is a valid instrument for evaluating school canteen policy compliance at a population level. Two papers describe how mobile technologies can be used in data collection: Heesch and Langdon¹⁹ show the potential and limitations of GPS data for measuring the effects of infrastructure developments on cycling; and Engelen et al.²⁰ offer a Brief Report on ecological momentary assessment via smartphones as a method for assessing impacts in worksite health promotion strategies.

Complementing the methodological papers in this issue are four reports of evaluations of complex health promotion initiatives, involving multiple settings and partners, and action to bring about change across several levels (i.e. individuals, organisations, environments and/or policies). Kearney et al.²¹ apply systems theory to the evaluation of a whole school approach to violence prevention that examined student attitudes and skills, class room practices and curricula, and school polices and culture. Genat et al.²² report the findings of the statewide Aboriginal led Victorian Aboriginal Nutrition and Physical Activity Strategy, which set out to build the capacity of Aboriginal and non-Aboriginal professional, organisational and community participants to take action on these issues. Two further papers illustrate how evaluations provided insights to improve the implementation and impact of worksite health promotion programs: Khanal et al.²³ report on a developmental evaluation of the Get Healthy at Work program in over 3000 worksites in New South Wales, while Grunseit et al.²⁴ describe how a meso-level evaluation of the Health Workers Initiative in seven Australian jurisdictions revealed critical factors affecting the translation of a national initiative into state-specific programs.

The imperative for high-quality evaluation in health promotion, and the commonly reported barriers to achieving this, have stimulated consideration of systematic approaches to building evaluation capacity. The NSW Health Department has invested significant resources in this area, and the final Brief Report in this issue describes the NSW Health approach to building research and evaluation capacity in population health.²⁵

The range and quality of contributions to this special issue demonstrate the continued learning that is taking place across the health promotion field that can strengthen evaluation design and methods, and the capacity of practitioners and agencies to implement this in a systematic way. We hope that you find new insights within these pages that are of value to your area of work.