Influencing population health performance: feedback from managers, population health staff and clinicians on the NSW Population Health Standards for Area Health Services

In the health sector, standards have been developed for individual practice, programs and organisations, and are used to assess performance and encourage its improvement.¹ Standards Australia defines a standard as ‘a published document which sets out specifications and procedures designed to ensure that a material, product, method or service is fit for its purpose and consistently performs in the way it was intended’.² A standard ‘encodes within it knowledge about how to’ and is used to transfer that knowledge into practice.³

The NSW Department of Health commissioned a series of projects to evaluate the NSW Population Health Standards for Area Health Services introduced in late 2005. One of these projects determined the extent to which the Standards were reflected in area health service performance agreements.⁴ This paper presents another of these projects and examines the Standards’ potential value from the perspective of the area health service staff who would be required to apply them.

Staff participated in the process either through interview or group discussion.

In order to gather a wide range of views, staff were purposively selected from Greater Western Area Health Service (GWAHS) executive and non-executive tiers with different work roles (managerial, population health), work settings (hospital, community health, population health) and directorates (Population Health, Planning and Performance, Clinical Operations) represented. Staff were contacted directly and provided with information about the Standards. They were informed that their participation in the process was voluntary and that responses would be de-identified to maintain confidentiality. There was the opportunity to discuss any concerns before verbally consenting to participate. Sixteen of 18 people contacted (nine women and seven men) agreed to be interviewed. All interviews were conducted by one of the authors and audiotaped. Interviews ran for 30–45 minutes.

After a pilot of two interviews, a list of open-ended questions was developed and refined. The questions sought feedback on: the self-assessment process; the Standards’ potential benefits and risks; factors that would help or hinder implementation; and how the Standards could be improved. The questions were an initial prompt for further responses and discussion. Pilot interviews were not included in the final analysis.

As the views of clinicians had not been sought in the interviews, a group discussion was held with eight clinicians working in hospital and community health settings. Participants were given information about the Standards before the session and were asked to consider the strengths and weaknesses, and factors that would help or hinder implementation. One of the authors facilitated the session and another took notes. The session ran for approximately 60 minutes.

Data analysis occurred concurrently with data collection. Audiotapes of interviews were transcribed for meaning, rather than verbatim. An initial reading of the transcripts identified the main points, which, along with pertinent quotes, were documented on an interview summary sheet. Transcripts were then coded by a single person. As further interviews were transcribed and coded, similar codes were collapsed into categories. Categories that recurred across interviews were noted as potential themes.

For comparison, two transcripts were coded by another author. Differences in coding and interpretations were discussed to reach agreement about categories. These categories, with associated quotes, were further discussed with all authors to reach agreement on themes.

Notes taken during the group session were read to identify the main points. Points in addition to or contrary to those gained from interviews were noted.

The self-assessment took place in 2006 during a time of significant organisational change. Participants noted the difficulty in using the Standards as a self-assessment tool in this context.

There was uncertainty among participants about how narrowly or broadly the Standards were to be applied: too narrow and the Standards have little relevance outside population health circles; too broad and the practicalities of data collection become a problem in completing the self-assessment task.

For self-assessment to impact on performance, there needs to be a way of taking action. For some participants, the results of the self-assessment would ‘drive some of the change’ and provide ‘a focus to enable us to work on specific strategies’. Others were more sceptical about the benefits of an audit. Future benchmarking with other area health services was seen as helpful.

Everyone interviewed reported that the Standards had potential benefits (Box 1).

All interviewees discussed possible risks or adverse consequences; in particular, that assessment using the Standards could become a bureaucratic process removed from everyday practice.

Staff involvement in developing and implementing the Standards was seen as crucial, and requires a good understanding of the tool and its purpose. One participant stated that the interview process itself had been educative. Participating in the process had helped their understanding of why the Standards had been developed and given them the opportunity to think about the potential value to their own practice. Others saw implementation as part of a bigger change, with one participant commenting that:

some of the changes that are required are about revolution, about a complete change in the way we do business and that requires a broader debate, a more inclusive debate, and this tool would assist that debate but you need general practitioners, specialist doctors and patients and community members using the tool, to get that sort of focus.

Successful implementation hinges on an organisation being orientated and committed to population health approaches. Most participants strongly expressed the idea that leadership and clear direction with tangible outcomes and rewards were needed along with links to other initiatives.

A recurring idea in the interviews was how to use the Standards in a way that demonstrates their practical purpose:

Being real, not being bureaucratic and ticking boxes so you can demonstrate that it [the Standards] is being treated seriously and that it is part of our core business.

Issues were raised about the capacity of the area health service to implement the Standards, including the need for resources and appropriate data collection systems (Box 2).

The language and jargon used alienated staff who were not specialists in population health and were not ‘applicable to people on the ground’.

To rate performance, the current form of the Standards uses four levels of achievement, from A (highest level) to D (lowest level). This system was not well supported. Level A was perceived to be:

unachievable and unhelpful as being a gold bar set up so high…it wouldn’t actually help to raise the standard of the work that you were doing.

Level D was thought to be equally unhelpful:

I think…it does not acknowledge any good work that is done…it basically says whatever it is you’re doing isn’t worth counting. I don’t think that’s a reasonable way of encouraging staff involved to take on board the Standards.

Several participants thought greater emphasis should be placed on equity issues and tackling upstream determinants of health. The health of Aboriginal and Torres Strait Islander peoples was seen by some as missing altogether, and hence detrimental to presenting the health of Aboriginal and Torres Strait Islander peoples as a high priority.

Feedback from the group discussion with clinicians was consistent with the interviews. The Standards in their current form were not seen as appropriate or accessible for clinicians. There was concern about the impact the Standards would have on workloads and how to manage any extra work. Clinicians understood the importance of engaging with communities about population health and thought that the Standards would help change expectations about health services and encourage communities to value evidence-based approaches.

Clinicians also wondered how these Standards related to other standards, including professional standards, and advised that the various reporting requirements would not be compatible. More detail about group discussion results is available in the evaluation report.⁵

The Standards were seen as a step forward in helping area health services assess and improve their own performance in population health. As an assessment tool, further modification is needed, especially to encourage ‘buy-in’ from managers and clinicians who are not working in specialised population health roles.

Feedback on the Standards was obtained from only a small number of staff in one rural area health service. Nearly half of interviewees worked in the Population Health, Planning and Performance directorate and had been directly involved in the self-assessment process. These staff members were therefore familiar with the Standards. Greater involvement of staff from outside this directorate may have elicited different findings. Nevertheless, there was consistency in responses between staff of the directorate and other staff.

This study took place during a time of significant change when all area health services in New South Wales (NSW) were undergoing restructure. GWAHS, an area health service with unique issues for implementation in terms of its geographical spread, was formed by merging three previously separate area organisational cultures. In these circumstances, staff interviewed may have been particularly aware of organisational barriers that needed to be overcome to allow successful area-wide implementation.

The Standards need to have credibility with area health service staff if they are to be successfully implemented and in turn influence population health performance. While there was general support for the idea of standards in population health, there was a strong feeling that population health should be central to the organisation’s business for the Standards to have credibility. The Standards were seen as potentially helpful in raising the profile of population health and setting goals for population health action.

Standards, as ‘encoded knowledge’ need to be ‘decoded’ easily. The current version of the Standards is written from a specialist population health viewpoint and does not translate well into what other area health service staff do on a daily basis. Area health service staff require time to engage with and extract meaning from standards expressed in plain language, and to decide how to use that knowledge to improve performance in their own setting.

The Standards as a self-assessment tool can identify current strengths and weaknesses, and identify where improvements could be directed. Those interviewed sought greater clarity about how area health services could take those next steps. Researchers note the lack of evidence-based tools to help improve performance in the population health context and the need for a science base that can ‘support accurate and reliable assessments of the practice of public health at local, state and national levels’.⁶

To effect these changes will be a major undertaking and that requires widespread support across the organisation and from outside, including support from the broader NSW community. The NSW Department of Health could assist area health services in a variety of ways, for example, through: revision of relevant policies to strengthen their population health orientation; engaging with other jurisdictions to develop a nationally coherent approach; and providing statewide co-ordination and resources where required.

The Standards show promise as a tool for area health services to assess their ability to deliver services in line with a population health approach. While it would be easier and less resource intensive to confine the use of population health standards to population health staff, if applied across an area health service, they offer greater potential to break down ‘silos’ between clinical and population health disciplines and harness more local expertise to tackle issues affecting the health of populations.