Evaluating Aboriginal and Torres Strait Islander health promotion activities using audit and feedback

Additional keywords: evaluation, Indigenous population, quality improvement, primary health care.

Health promotion for Australian Aboriginal and Torres Strait Islander communities (respectfully referred to hereafter as Indigenous) and their people has generally had limited efficacy and sustainability (Demaio et al. 2012). Effective health promotion is important because there are large health inequalities in Australia, with Indigenous Australians known to suffer the poorest health of any population group (Australian Institute of Health and Welfare 2014). Access to healthcare and differences in lifestyles (e.g. smoking, alcohol misuse) are important determinants of health inequalities, but so are the circumstances in which people are born, grow, live and age, and the extent to which they have control over these circumstances (Marmot 2011). Many of the causative factors of health and ill health lie outside the health system and, to address these, a range of measures that extend beyond the health sector is required. To this end, comprehensive primary healthcare (PHC) is recognised as central to providing a multidisciplinary framework that can interface with other sectors and tackle Indigenous disadvantage (Marmot 2011; Donato and Segal 2013), with health promotion as a core function (Tilton and Thomas 2011). Within this complex context for health care delivery, evaluation of health promotion activities is imperative.

Indigenous PHC services have been identified as exemplary models of comprehensive PHC (Wakerman et al. 2009); however, many struggle to deliver effective health promotion. In particular, typically they do not have the expertise to design robust evaluations or ready access to tools to evaluate health promotion activities (McCalman et al. 2014). When evaluations are not done or not done well, this has implications for determining the impact and health outcomes of Indigenous health promotion activities (Mikhailovich et al. 2007).

Across Australia, Indigenous PHC services are using a popular continuous quality improvement (CQI) technique, known as audit and feedback (A&F), to evaluate and improve the delivery of clinical care (Wise et al. 2013). A&F is a process of reviewing care against explicit criteria (such as guidelines for recommended care) and providing feedback on performance to practitioners (Jamtvedt et al. 2006). Standardised audit tools are used to collect information related to the explicit criteria from a representative sample of medical records. The gap between assessed performance and the set criteria provides guidance to identify improvement strategies. With repeated A&F cycles, the effectiveness of these strategies for improving the quality of health care can be monitored (Bailie et al. 2007a). An international systematic review has found A&F works because practitioners are prompted to modify their practice if they are given feedback that their current practice is inconsistent with recognised best practice (Jamtvedt et al. 2006). Although A&F has been successfully applied to improve clinical processes within Indigenous PHC (Bailie et al. 2007b; Gardner et al. 2011) and used globally in a range of PHC applications (Jamtvedt et al. 2006), we are unaware of any previously published evidence using A&F in health promotion, and in particular as a focus for evaluating Indigenous health promotion activities.

In this paper, we report on the feasibility and acceptability of the A&F approach in health promotion, and the use of a purpose-designed audit tool as part of that approach, to evaluate the design, implementation and evaluation of health promotion activities in Indigenous PHC services.

The 3-year feasibility study in using A&F to improve health promotion was based on a CQI research program known as the Audit and Best Practice for Chronic Disease (ABCD) project. Drawing on A&F techniques, the ABCD project demonstrated that positive changes can be made to Indigenous PHC service systems, care delivery and intermediate health outcomes for the prevention and management of chronic illness care (Bailie et al. 2007b). For health promotion evaluation, an A&F approach introduces the concept of obtaining standardised information about key areas of best practice in health promotion; this is different to traditional evaluation approaches using process, impact and outcome evaluation. A&F typically involves a four-step process of: (1) audit; (2) analysis and interpretation; (3) feedback; and (4) strategies for improvement. A shift in thinking is required, from a focus on evaluating an individual health promotion activity to assessing (all or a sample of) health promotion activities at the organisational (or unit) level.

As part of the study, and consistent with A&F techniques, a standardised audit tool was developed to capture information about criteria related to best practice health promotion (Fig. 1). We also developed a protocol to support data collection using the audit tool. The protocol provides rationale for the collection of information (or data) in a standardised way, explains the audit questions and provides examples to facilitate data collection.

Fig. 1.  Best practice criteria for Indigenous health promotion.

Importantly, our A&F approach was guided by principles of participatory action learning (Israel et al. 1998) and emphasised the importance of engaging those people involved in service delivery as fundamental for achieving quality improvement (Bailie et al. 2007a). The involvement of leaders and champions, and staff participation in all parts of A&F, supports individual learning to achieve a shared understanding of required changes (Allen and Clarke 2013). Participatory approaches are aligned with capacity building approaches (Hawe et al. 1997) and principles of culturally appropriate health promotion for Indigenous Australians (Mikhailovich et al. 2007). Our research experience supports this alignment. We collaborated with research participants to simultaneously refine the four-step A&F process while building practitioners’ understanding of good health promotion practice. The participatory nature of workshop sessions (facilitated by Indigenous and non-Indigenous members of the research team) enabled Indigenous perspectives and understandings of health promotion to be shared and discussed in a safe space by cross-cultural health teams and incorporated into action plans.

A key feature of capacity building and participatory approaches is facilitation. In the context of using A&F to evaluate health promotion, facilitation enables a two-way dialogue between a facilitator and health service teams to build a shared understanding of recommended best practice (Harvey et al. 2002) and how this compares with the local health promotion practice. Facilitators can include health promotion practitioners, CQI facilitators or those in similar roles who are able to lead, implement and operationalise A&F processes.

The research was conducted in collaboration with four PHC services in the Northern Territory (NT) over two A&F cycles. Three of the PHC services were governed by a board of elected Indigenous community members who provided guidance on the provision of health services to people in their respective communities (community controlled); one service was managed and operated by the NT Department of Health (government service). Each PHC service employed multidisciplinary teams of between five and >50 staff, including nurses, allied health, doctors and Aboriginal health workers. Two PHC services were located in communities with populations of >1000 people (but <5000); one PHC service was located in a community of >5000 people; the fourth was located in a community with a population of <500. Three of the four participating PHC services were sole providers of PHC in their respective communities. Although the PHC services varied in size and governance, they all served populations with a high burden of chronic illness and injury, and experienced considerable staff turnover and high demand for acute and emergency care services. These service-level factors were considered when planning and facilitating A&F for health promotion.

This research project was granted approval by the Human Research Ethics Community of the NT Department of Health and Community Services and Menzies School of Health Research (07/01).

An iterative and participatory process was used to develop an audit tool drawing on: (1) a review of existing tools, frameworks and best practice literature; (2) consultations with key Indigenous and non-Indigenous stakeholders representing PHC and health promotion practitioners in the NT; and (3) field testing at participating PHC services. Our definition of ‘health promotion’ follows the Ottawa Charter principles to advocate, enable and mediate (World Health Organization 1986) and recognises that to be effective, Indigenous health promotion must be culturally appropriate, community controlled and based on the goals of Indigenous communities (Mikhailovich et al. 2007). Best practice criteria for health promotion were identified in order to inform audit tool development. Three principles guided our decisions in the identification of criteria: (1) the criteria needed to reflect key design phases of planning, implementing and evaluating health promotion activities; (2) there was evidence that their application could contribute to improving health promotion quality and effectiveness; and (3) they had potential to provide PHC staff with information about changes in their HP practice over time, when A&F cycles were repeated.

The resulting health promotion audit tool was designed to capture information about the following five best practice criteria: (1) comprehensive planning; (2) systematic targeting; (3) community participation; (4) partnerships; and (5) evaluation (Fig. 1). (The audit tool and associated data collection protocol are available from One21seventy, The National Centre for quality improvement in Indigenous primary health care; http://www.one21seventy.org.au, verified 6 July 2014).

As illustrated in Fig. 2, and consistent with international A&F practice (Jamtvedt et al. 2006), our A&F approach includes four steps: (1) the audit, (2) analysis and interpretation, (3) feedback and (4) strategies for improvement.

Fig. 2.  Audit and feedback approach.

The research team coordinated a communication plan with managers at each of the four health services to prepare for health promotion audits. A pre-audit information sheet was sent out 1 month in advance of the agreed date to assist with the collection of documentation of health promotion project activities and other potential sources of information for the audit. To help health service staff identify health promotion activities to audit, we advised using the four categories of health promotion interventions frequently used in the literature (Victorian Department of Human Services 2003): (1) health education and skill development; (2) health information and social marketing; (3) community action strategies; and (4) settings and supportive environment. Follow-up contact occurred 2 weeks and again a few days before the audit day. A manager, health promotion officer or quality improvement facilitator worked closely with the health team to assist with the pre-audit preparation. On audit day, the research team facilitated a process through which health promotion project records were divided among the health centre team, who worked in pairs (rather than individually) in order to encourage reflective discussion while auditing. The audit tool was used to review available project records, with the process guided by a data collection protocol and supported by the research team facilitators.

We found that having a cofacilitator from the local health service, such as a manager, CQI facilitator or health promotion coordinator, was important for local ownership and implementation of the A&F approach. Cofacilitation helped support, guide, coordinate and communicate the A&F approach within the PHC service.

Health service teams reported the audit preparation and process useful in several respects. Although the Ottawa Charter can provide guiding principles for health promotion, some PHC service staff had found it challenging to operationalise health promotion in terms of tangible actions; sorting health promotion activities into four categories helped teams identify their health promotion activities. Furthermore, many participants found the audit tool useful in raising their awareness and understanding of best practice health promotion and said the audit tool provided a ‘check list’ for planning health promotion activities. Several PHC staff reported that, by doing the audit, they had realised a lot of health promotion work had been done but not documented. As a result, the limited records available for auditing had not accurately reflected health promotion efforts to that point and could not adequately inform ongoing health promotion planning.

This step was a research process in which the research team collated the combined audit data from the four PHC services, using spreadsheets to generate results. The team sorted, analysed and interpreted the audit results using quantitative (numbers/statistics) and qualitative (words/stories) methods. Audit results were transferred into an illustrated report for each participating health service. Quantitative data were presented as pie charts, bar graphs, stacked bar graphs and tables. The qualitative data were presented descriptively, along with a brief explanation of what the evidence says about best practice for each health promotion criterion. For example, audit results for ‘partnering’ were presented with an explanation of best practice partnerships for health promotion (including the benefits of partnering and how the formation of partnerships and choice of partners may influence audit results at their health service). Reports included a diagram of a health promotion planning cycle (illustrating the five best practice criteria) and a discussion guide for local facilitators to use, ahead of the research team’s feedback visit. The research team put high priority on user-friendly reporting, recognising the time constraints on the PHC service staff engaging in A&F processes and the varied professional backgrounds of participating staff. The small number of health promotion activity records kept by the participating health services and available for auditing made it difficult to represent data meaningfully. However, we found the A&F processes enabled discussion about these limitations and motivated practitioners to increase documentation of health promotion activities.

In consultation with PHC service managers, the research team visited each health service to feed back results to staff, ensuring that adequate time was quarantined to enable reflection and discussion. On the day of feedback visits, the illustrated reports were shared between staff and discussion of results and further interpretation was facilitated by the research team. The interactive sessions raised questions and provided opportunities for staff discussion. For example: ‘What do the audit results tell us about community participation?’, ‘Did community involvement occur throughout a health promotion activity, or in particular phases?’, ‘Are there key groups in the community who could be more involved?’. The facilitated discussion enabled staff to identify strengths and gaps in their practice and to think about priority areas for improvements.

We found an illustrated report was an important resource for providing a narrative or story that emerged from the audit data. It stimulated staff interest through the use of illustrations, diagrams and graphs, and a table structured around the five best practice criteria for health promotion, to reflect what was happening in the local context. Staff took ownership of the report, which also served as a record of the results and discussions.

The combination of local and external facilitator knowledge was important for interpreting results and exploring the relationship between current practice and best practice. Structured and facilitated interpretation and feedback sessions, with quarantined time for staff discussion and reflection, were also important for engagement with the data and discussing implications for future health promotion practice.

Following the feedback discussion, the research team facilitated the development of a local action plan for health promotion. Action plans identified three priority areas for improvement. Each of the priority areas identified was further developed to determine what would happen (strategies), who would do what, time frames and resources required to implement the planned improvements or changes.

We found this step in the A&F approach challenging; it required a lot more support from the research team external facilitator to assist with identifying the most appropriate priority areas for improvement action. Some health service teams were more proactive in developing action plans than others. Three monthly follow-up support was an important strategy to support local teams to implement their action plans and to provide additional advice and resources. For example, the research team provided planning templates, Internet links to websites for information about best practice and opportunities to discuss and reflect on what was happening in local contexts.

An A&F approach in health promotion provides a useful tool and strategy for supporting practitioners to evaluate the design of local health promotion activities in Indigenous PHC services. The A&F processes engaged practitioners in using an audit tool and protocol to capture locally relevant documented information, which was used to plan strategies for practice improvement. A strength of the A&F approach is its emphasis on participation in, and facilitation of, a quality improvement dialogue. We found that facilitation of A&F and a quality improvement dialogue by the research team in the local context enhanced learning through hands-on application, and enhanced opportunities for practitioners to extend health promotion networks beyond the health service. Provision of quarantined time at PHC services for A&F provided a dedicated learning space for practitioners to improve their knowledge and understanding of health promotion. Feedback of results, the identification of priorities and setting of improvement strategies to modify practice were critical in supporting individuals and teams with monitoring and assessing their improvement changes. The A&F approach and processes helped PHC services to determine roles and responsibilities for health promotion.

Studies have shown that a collaborative approach to assessment is a useful educational process with practitioners, as well as being a source of information for evaluating change (Hawe et al. 1997; Labonte and Laverack 2001). However, auditing health promotion activities can be time consuming and challenging due to the availability (and accessibility) of documentation as a source of data for audit and feedback purposes. This is because health promotion activities often have defined endpoints, are evolving in nature and vary according to their state of development and implementation. Furthermore, participating PHC services lacked systems for documenting, storing and communicating information about health promotion at their health service.

This paper describes the acceptability and feasibility of applying the A&F approach, but does not evaluate its effectiveness for improving health promotion. Some promising results have been reported (Percival 2014), but further research is needed to determine, more conclusively, the effectiveness of the A&F approach for improving Indigenous health promotion over time. The feasibility was tested and assessed in only four PHC services with prior familiarity with clinical CQI approaches; however, they varied in size and health promotion capacity. Hence, we consider the findings to be generalisable across Indigenous health promotion within PHC.

The researchers found a lack of tools or studies to draw upon when developing tools to support A&F in health promotion.

The present study found that the A&F approach was both acceptable and feasible for supporting practitioners to evaluate planning, implementation and evaluation of health promotion activities in Indigenous PHC services.

The use of an audit tool and A&F approach enables PHC service staff to reflect on their documentation and practice of health promotion, and to plan improvements based on this evidence. At the organisational level, health promotion A&F enables a shift from evaluation of individual activities towards evaluation of health promotion activities as part of a service’s comprehensive PHC framework, and may inform system changes at that level.

A structured A&F approach, a purpose-designed tool and the allocation of quarantined time for A&F processes at the health service level are important strategies. Facilitation plays a key role in supporting the engagement of practitioners in the practical application of A&F in the local PHC service context. It assists in building understanding of health promotion, as well as the confidence to identify health promotion improvement strategies and actions that are tailored to meet locally identified needs.

None declared.