Health policy analysis: a tool to evaluate in policy documents the alignment between policy statements and intended outcomes

Additional keywords: chronic disease, criteria, internal validity, policy formulation.

A health policy is a plan that steers the direction of investment and action designed to alleviate suffering, improve health care or prevent illness. It can be manifested as laws, bureaucratic edicts, practice guidelines or, more vaguely, simply as guiding principles.^1,2 Health policy makers are tasked with navigating a path between competing interests and demands to develop a pragmatic response to one or more health problems.² There has been increasing interest in evidence-based policy making, which strives to use the best available evidence to inform policy. Although evidence from research is not the only factor influencing policy making, it has considerable potential to contribute to effective health policy.³

Evidence from health policy analysis and evaluation can potentially increase policy impact and provide information that may assist with the allocation of scarce resources. Policy impact can refer to strategies of policy implementation (outputs), or changes in population health or the health of the individual, as is the case when clinical management policies are applied (outcomes).⁴ If a health policy can be shown to be successful in achieving its goals this may increase the likelihood of ongoing funding. Although there may be opportunities for funding without such evidence, perhaps as a result of political and economic factors, proof of goal achievement is additional enticement to the distributors of funding. Procurement of ongoing funding greatly improves the chances of successful policy implementation.

Detailed health policy analysis and evaluation assists policy makers to improve the chances of successful implementation of future policy by revealing opportunities where enhancements to policy documents may be made. Such enhancements may be added to future policy documents or potentially to the original documents if applied before the policy is finalised. Analysis of the internal validity of policy documents is one approach to achieving these enhancements. Internal validity, in this context, refers to the clarity and comprehensiveness of policy statements to reflect intended outcomes. Health policy documents do not always articulate intended outcomes optimally; policy writers are not commonly responsible for implementing the policy, and details that can affect the ease with which a policy is implemented and thus its success may not be known or may be overlooked.

The development of policy documents is one part of the policy process that enables goals, opportunities, obligations and resources to be recognised in a concrete form and, through careful analysis of the documents (policy document analysis), the extent to which a policy adheres to certain principles, such as stakeholder and legislative support and goal clarity,^1,5,6 may be ascertained. Policy documents should be referred to implementers: (1) before action, for guidance on how best to ensure that the policy will be implemented in a way that fulfils its goals, and (2) during the implementation phase to monitor progress and ensure that the process ‘stays on track’. Policy documents should also be utilised during the evaluation phase to reconcile policy goals with outcomes and to allow minor iterative or more substantial changes to be made to future policies, thereby increasing their impact.

Policy document analysis is one comparatively straightforward method to appraise the extent to which the policy conforms to influential principles critical for successful implementation. It can be aided by carefully considering and mapping the linkages between closely related policy documents, particularly for complex initiatives. Mapping such linkages may also promote more coherent future policy development.

Complex health issues, such as chronic disease, involve complex initiatives in their management. For instance, the successful management of cardiovascular disease, and its common comorbidities, obesity and diabetes, requires interplay between several health policy areas. The need to develop effective chronic care policies and programs is worthy of our attention. This group of diseases is currently responsible for ~70% of the total burden of illness and injury experienced by the Australian population and this proportion is expected to increase to ~80% by 2020.⁷ The ageing population and increasing prevalence of risk factors for chronic conditions are set to increase mortality and morbidity from chronic diseases.^8,9 These trends are putting increased pressure on health systems. In 1999–2000, 17% of hospital admissions were for patients with chronic respiratory and cardiovascular diseases, diabetes and cancer at a cost of $1.1 billion in NSW.¹⁰ Chronic care is the major component in the rising health care cost in Australia and is likely to consume ~75% of the health budget in the future.¹¹

The Chronic Care Program (NCCP), established in 2000, is the first major policy attempt to address the broad spectrum of chronic illness care in the entire state of NSW (Population: 7 million) in Australia.¹² It is a policy-rich initiative (Fig. 1) (that is, with many policy papers) that aims to improve the quality of life for people with chronic illness and reduce inappropriate admission to intensive acute-care hospitals of people with episodes of deterioration for which care in the community is optimal. The policy is a major initiative of the NSW Department of Health, which oversees authorities responsible for the provision of public hospital and much community-based health care in the state.¹²

We demonstrate here the utility of a set of predefined criteria for use in retrospective policy document analysis, which may also be of use in making future health policies. We chose to evaluate policy documents on chronic illness because of its increasing burden on the community and because it has been the subject of intense policy activity recently in NSW. The paper is not intended to be an analysis of how successful the implementation of the NSW Chronic Care Program has been. It is intended to be an exploration of a tool for analysis of the internal validity of policy documents, to appraise the extent to which policy statements align with intended outcomes.

Since policy is often about reconciling different value perspectives, the role of evidence is not the only thing that shapes a policy. Scientific and technical evidence has been described as one of three forms of evidence that commonly contribute to policy,¹³ in addition to political knowledge and professional experience.

We chose to evaluate the NCCP for alignment between policy statements and intended outcomes, using criteria validated by Rütten et al. These criteria were derived from von Wright’s ‘logic-of-events’ theoretical framework.^14,15 This predefined set of criteria was chosen because it provides an easily understood and persuasive connection between policy determinants and policy outcome and has been validated as a useful tool in analysing health policy.⁴

Several models have been put forward to describe the complexities of the policy process.^16,17 There has been no single model whose utility prevails above all others in all contexts. Instead, the varying strengths and weaknesses of each make some more suitable than others in different situations. The health policy process is rarely as undemanding as the linear models of policy development described in the literature, which outline a process of agenda setting, decision making and implementation of solutions that is followed in a linear fashion. The linear model assumes that the policy development process is a rational, objective, balanced and analytical one. It has, however, previously been described as overly simplistic because the policy process is rarely as straightforward as this model would suggest.¹⁸ On the other hand, the ‘garbage can’ model suggests that decisions are not always the result of logical processes and instead decisions arise from randomly crossing paths taken by participants, opportunities, problems and solutions.^16,17 However, in advocating ‘organised anarchy’, it does not examine the effects of structure in organisation in producing order and increasing productivity.¹⁶

An Australian policy cycle model proposes that policy development proceeds through the following stages: issue identification, policy analysis, policy instruments, consultation, coordination, decision, implementation and evaluation.¹⁹ Policy cycle models have previously been described as idealistic and do not recognise the role that opportunity and circumstance play in driving the process.²⁰ In contrast, the von Wright ‘logic of events’ model describes action as a result of the interaction of a combination of determinants including wants, abilities, duties and opportunities. Wants and duties act as an impetus for action. Abilities can be the constraining force that limits the extent of action and opportunities for future action change as a result of each prior action. A strength of the ‘logic of events’ model is that it recognises that the relationship between determinants is not static and varies as situations change.¹⁴ Similarly, the relationships between influencing factors in policy-making vary with the circumstance.

Although policy analysis frameworks exist that focus on areas such as content, process and feasibility of implementation, a framework for analysis of policy documents does not.^20,21 Rütten et al. adapted the logic of events model within the context of health policy to develop a framework for health policy analysis.^4,15 In this framework, goals, resources, obligations and opportunities form the determinants of output of health policies. Rütten used this framework to examine health promotion policies,⁴ and its utility is also demonstrated in other literature.²² Its focus on policy formulation and implementation enables the framework to be adapted for document analysis and evaluation of other health policies.

As set out below, we added 16 new criteria (indicated in italics) to those of Rütten et al. after literature review and consultation with experts in the field (see acknowledgments for the list of experts).^23–25 Some of the criteria have strong relevance to policy context (e.g. political opportunities) and, as a result, were difficult to assess by examining policy documents. They have been excluded from the analysis and marked (**) in Box 1. The need to amend criteria is a reflection of the complexities of the non-linear relationship between policy documentation and process.

Box 1.  Criteria for analysing policy documents

The following is an explanation of why amendments were made to the criteria of Rütten et al.⁴:

1. Accessibility. Document accessibility may be a facilitator or barrier to usefulness and implementation of policy. Senior executives, health service planners, health service managers, those responsible for direct health service delivery, general practitioners, private providers of care for people with chronic disease and the general public were identified as the intended audience for this program.²⁶

2. Policy background. Policy background encompasses the consideration of scientific results and so the Rütten criterion ‘scientific results demand the action’ has been incorporated into this section. Sources may be of different types: authority (e.g. persons, books, articles), quantitative or qualitative analysis, deduction (premises that have been established from authority, observation, intuition or all three).^1,25

3. Goals. Detailing precise change mechanisms may provoke unnecessary resistance to amendable details and deny achievement of the final goal.²⁷ Goals should demonstrate consistency, which is associated with dependability. External consistency refers to observations made in other situations that support the policy proposal.²⁸ Internal consistency refers to inferences logically drawn from the available information.²⁹

4. Resources. Rewards or sanctions for spending the allocated financial resource on other programs can affect the likelihood of success of a policy. Organisational capacity criteria were amended to enable assessment.

5. Monitoring and evaluation. Independent evaluation strengthens the analyses’ credibility. Data collection before and after implementation also increases the credibility of the evaluation.

6. Political opportunities. Assessment of political opportunities is difficult using document analysis. Thus, this criterion was excluded from analysis.

7. Public opportunities. Rütten’s original criteria are difficult to assess using document analysis and have thus been excluded.^1,30,31 Other aspects of public opportunities such as stakeholder involvement will be assessed.

8. Obligations. One of the links between goal setting and successful implementation is the development of explicit objectives.¹ It follows that part of this is the clear specification of the obligations of various implementers.

Background and evaluation documents related to the NCCP were located using relevant health service websites (e.g. NSW Health website) and other search engines (e.g. Google). In addition, informal inquiries to field experts were pursued to locate material not listed in mainstream databases. Documents relevant to chronic care and the NCCP were then mapped to demonstrate the relations among the documents and to facilitate the analysis and searching (Fig. 1).

Analysis was conducted of the following documents Improving health care for people with chronic illness – a blueprint for change 2001–2003¹⁰; NSW Chronic Care Program: Phase Two 2003–2006³²; NSW Chronic Care Program Phase Three: 2006–2009²⁶; NSW Chronic Care Program 2000–2003: Strengthening capacity for chronic care in the NSW health system³³; NSW Chronic Disease Prevention Strategy 2003–2007³⁴; and two other relevant documents³⁵ (Fig. 1). The criteria in Box 2 were used to evaluate their internal validity. The documents were selected because they are major policy documents for dealing with chronic illness in NSW.

Box 2.  Summary of policy documents analysis of New South Wales Chronic Care Program documents

Two of the authors, KC and MM, performed the literature search, mapped the extracted relevant documents and critically appraised them against the criteria in Box 2. To ensure consistency and to minimise subjectivity, KC and MM cross-checked their mapping and analysis of the documents and consulted with SRL if a difference in the analysis arose. The face validity of the amended list of criteria was assessed through consultation with a group of Australian and international field experts, acknowledged at the end of this paper.

Box 2 summarises the NCCP policy document analysis using the criteria described in the Methods section (Box 1).

Criteria were considered ‘Fulfilled/Strong’ if all the mentioned criteria were addressed, ‘Room for improvement’ if some criteria were addressed and ‘Not fulfilled/Weak’ if no criterion was addressed.

Core documents were accessible from the SWSAHS website.¹² The NCCP Phase 3, 2006–2009: NSW Chronic Disease Strategy was available on the NCCP homepage from January 2009. It was not available from the policy documents or NCCP homepage before this date during policy implementation. It is unclear how accessible the documents were for parties responsible for direct health service delivery, general practitioners and private providers of care should they wish to have referred to the document during policy implementation. Although such access problems may have applied to the public, they probably did not apply to senior executives, planners and managers.²⁶

The policy states ‘The strategy outlines key developments in chronic care literature and practice’.²⁷ Authority, statistics and deduction were all used by policy writers in establishing the policy background, including references from Australian government departments,^10,36 peer-reviewed journal articles,³⁷ presentations³⁸ and statistics from sources such as the Australian Institute of Health and Welfare, World Health Organization and Centres for Disease Control and Prevention (USA), among others. Expert reference groups were involved.¹⁰ The ‘Partners in Health approach’ is an example of how the NCCP developed some programs based on ‘sound’ theory.²⁶

Goals are explicit and concrete, e.g. ‘reduce crisis situations and unplanned and avoidable admissions’, and quantitative (e.g. can be measured quantitatively) where possible and qualitative where not.³² The action centres on improving the health of the population. Employment of some of the methods used was based on literature (e.g. ‘The NSW Chronic Disease Strategy has built on the Kaiser Permanente/NHS model’²⁶) and external consistency (e.g. ‘Research and practical experience in North America and Britain are showing that…’).³³

Financial resources (e.g. ‘investment increased to support … action on CDP’,³⁴ ‘Recurrent funding of $15 million per annum has been allocated’),³² human resources (e.g. ‘over 200 staff were employed’),³² and organisational capacity (e.g. ‘providing robust … infrastructure and services to underpin program’) were addressed.³⁹ It is unknown whether the resources were sufficient.

The policy indicated mechanisms ‘for monitoring and evaluation of the NSW Chronic Disease Strategy (2006–2009)…’,²⁶ and ‘development of systems to improve the monitoring … using an agreed and standardised methodology’.³⁹ Although the NSW Chronic Care Collaborative (NCCC) 2005 notes ‘an independent evaluation of the NCCC undertaken by the Centre for Health Services Research at Westmead’, it was not clear who was responsible for other evaluations. Data were collected at baseline and at follow-up. Results such as a ‘70% reduction in hospital admissions’ suggest that policy change produced an effect. It appears that the time between baseline and follow-up (five years) was sufficient to allow the effects of policy change to become evident. However, the methods used in evaluating quantitative and qualitative achievements have not been mentioned or referred to.³³ Other factors that could have produced the change were not identified.³³ A ‘three lens’ approach may be useful for further analysis of these results.¹³

The importance of stakeholder involvement is acknowledged: ‘Stakeholders include … general practice … hospitals … non-government organisations…’, ‘…developed in close consultation with Aboriginal community members to ensure their appropriateness and appeal’.²⁶ The extent to which clinicians, community health services and general practitioners were consulted could be clearer; for example, how significant were the consultation(s) in shaping policy? How regular and at what point(s) of the policy process were consultations obtained?

The obligations of the various implementers are specified with potential for greater clarity. The policy documents analysed indicated that Key Performance Indicators were developed and that groups responsible for policy implementation were to report progress. Access to these documents online proved difficult. The parties to whom implementers were held accountable were also specified. It is unclear whether there were any rewards or sanctions to reflect fulfilment of obligations.^10,26

Many of the tabulated criteria were met by the several policy documents that comprise the NCCP. Policy background and goals were the NCCP’s greatest strengths. Considering the project’s magnitude, there is no surprise that opportunities for improvement of the policy documentation were identified through the use of the modified criteria. Opportunities identified include accessibility, resources, obligations, public opportunities and monitoring and evaluation.

In light of the sheer number of documents relevant to the NCCP, anyone who did not already know of them all, and where they might be found, would have difficulty finding the documents and understanding their complex relationships. It is analogous to piecing together a jigsaw having never seen the complete picture. Document mapping proved to be beneficial for this project and would be useful for other projects of a similar scale in facilitating an initial analysis. The use of grey literature provides policy context and implications not found in the published literature, which can add depth to the policy analysis.⁴⁰

The criteria may be viewed as a ‘check-list’ that enables policy makers to review how closely policy intentions are reflected in their documents. More specifically, the criteria can reveal how policy documents may be amended to take into account lessons learned from successes and failures experienced. A strengthened policy can potentially increase the likelihood of successful implementation.

Appropriate goal setting can strengthen a policy. The goals were clear in their intent and did not attempt to prescribe in detail what the change must be, fulfilling the policy analyses criteria. However, the mechanism used to achieve the desired goals could have been clearer. Document succinctness leads one to suspect that details such as goal mechanism and data or statistical analysis have been explored in greater depth elsewhere, such as in internal documents, and intentionally omitted from the policy documents for the sake of clarity and brevity. It is also possible that such details have not been explored or are not intended for public access. However, citing those materials would be helpful for external evaluators.

The financial and non-financial resources are addressed in the policy documentation; however, improved accessibility to greater detail about monitoring and evaluation may facilitate analyses of whether resource availability was sufficient. This information is potentially useful in future planning and allocation of resources to fulfil obligations, which may be facilitated by rewards and sanctions. Lack of rewards and sanctions may raise issues such as the tempting possibility that funds allocated to policy implementation may be reassigned to other areas of need not specifically targeted by the policy, thereby reducing the likelihood of successful policy implementation.

Finally, the perceived credibility of the policy documents is enhanced by the recognition of challenges and limitations, e.g. ‘Evaluating the impact of priority health care programs on carers as a defined group proved challenging for most local programs’.³³

This study has attempted to evaluate the main policy documents governing health service provision for the chronically ill and to develop a set of criteria for retrospective assessment of the internal validity of policy documents in an Australian context. Such criteria could also be used prospectively to review policy proposals in the evaluation phase and to strengthen health policies in the development phase. Expert opinion and consultation is valued in criteria development and well defined criteria have evolved in this manner.^41,42 This is an ongoing endeavour and we invite comments from scholars in the field to further develop our method.

The NSW Chronic Care Program policy papers have some strength when assessed against a set of analysis criteria. Document mapping proved to be vital in obtaining a complete picture of the policy documents pertinent to the NCCP. Document analysis can identify where policies are not consistent with their intended outcomes. A validated policy document analysis tool for Australia can be a valuable instrument.

The authors declare that they have no competing interests.