The development of a streamlined, coordinated and sustainable evaluation methodology for a diverse chronic disease management program

In 2001, the Victorian Department of Human Services allocated funding to the Hospital Admission Risk Program (HARP) in an endeavour to arrest the growth in emergency department presentations. The HARP model targeted clients at risk of hospitalisation and provided a thorough assessment, planning, management and monitoring service delivered by a multi-disciplinary team. Fourteen diverse projects, from five auspicing agencies, were funded in the Northern Region of Melbourne. Initially projects were functionally independent and relatively isolated, with different models of care, target groups, and management structures. Evaluation of the projects was inconsistent, and in many cases lacking altogether.

As HARP progressed, the Department of Human Services commissioned an evaluation and the outcomes of this evaluation were released in 2005.¹ The main findings were that HARP was successful in reducing hospital demand and in building relationships across the care continuum. The Department of Human Services decided to mainstream HARP, with a commitment to ongoing funding. Consequently, HARP projects from the same health region were required to form a single program with unified governance and reporting.

In 2006 projects in the north of Melbourne moved to operating as one Northern Alliance HARP–Chronic Disease Management (HARP-CDM) Program. The 13 remaining services, from four auspicing agencies, were integrated under a single program with three ‘streams’: Complex Needs, Chronic Disease and Psychosocial Needs. In parallel with this came several changes to the collection and storage of patient information – the largest of which were the move to the HealthSmart patient management system and mandatory reporting to Department of Human Services on the Victorian Integrated Non-Admitted Health (VINAH) minimum dataset. In mid 2006, the Northern Clinical Research Centre, based at The Northern Hospital saw this as a unique opportunity to align the evaluation of HARP-CDM within an integrated patient management system and mandatory data collection requirements.

Integrated data collation and reporting was expected to facilitate evidence-based policy and planning by the HARP-CDM governance group. At the individual service level, comprehensive evaluation was expected to support service sustainability through ongoing quality improvements, increased project efficiency and the development of innovations in evidence-based care.

The objectives of this project were to develop a system-wide evaluation approach for the Northern Alliance HARP-CDM program that would:

• Contribute to evidence-based policy and planning at a regional and state level.

• Support local, clinician-led improvements and innovations in patient care.

• Maximise return on data collection and reporting with minimal burden on clinicians.

• Assist with the integration and coordination of services through performance monitoring and benchmarking.

• Generate further research questions and contribute to the evidence on disease management.

All existing, routinely collected administrative data items (especially emergency department presentations, hospital admissions and financial) and the VINAH minimum dataset items were audited by the evaluators. The VINAH minimum dataset contains 52 fields built around the client’s ‘episode of care’ within the HARP program. Comprehensive data including demographic, referral and service activity information is collected to describe each ‘episode of care’.

Following the audit the evaluators developed a list of analyses that could be performed with the available data. These analyses were compared with the HARP-CDM program objectives and evaluation gaps were identified. Measures of client specific, generic and disorder outcomes were identified as missing from the evaluation dataset, and a strategy to include them was developed.

To measure quality of life broadly across the entire Northern Alliance HARP-CDM program, the evaluators sought a validated, generic tool that calculated utility scores and was sensitive to a range of patient conditions and care models. The Assessment of Quality of Life (AQoL) tool was selected and introduced to the evaluation dataset. The AQoL² is a generic quality of life instrument and has tested as good or better than instruments commonly used in measuring outcomes in stroke, coordinated care, influenza, cochlear implants, population monitoring and elderly groups. The AQoL provides both health related quality of life information in the domains of ‘Illness’, ‘Independent Living’, ‘Social relationships’, ‘Physical senses’, ‘Psychological wellbeing’ and a total AQoL value. In addition to these health related quality of life outcomes, the AQoL may be summarised to generate utility values, thereby enabling the generation of Quality Adjusted Life Years and facilitating economic analyses.

The evaluators worked with the different teams to identify service specific objectives related to client outcomes. Once service-specific objectives were agreed, tools and measures which would quantify these outcomes were selected. Between one and three outcome measures were selected for each specific service. Like services were encouraged to select the same outcome measures so that comparisons between programs and data pooling would be possible. For example, the two diabetes services selected the same measurement to assess the impact of their services on blood glucose control.

The addition of client outcome measures to the VINAH, financial and hospital utilisation data completed the HARP evaluation dataset. All evaluation data have been embedded within HealthSmart so that data collection and storage are integrated with clinical practice, uniform and streamlined. Data are linked via the existing unique patient identifier (unit record number). Several automated reports are being developed to provide the clinical teams and administrators with information about client demographic profile, service activity, client outcomes, and cost-utility. These reports will provide summary data at three levels: overall program, the three clinical streams and each individual service. This provides managers, teams and funders with the ability to periodically review individual service components and the performance of the program as whole.

Ethical approval for the waiver of individual client consent, data collection, storage and analysis was received by the Northern Health Human Research and Ethics Committee.

Implementation of the entire evaluation dataset and its integration into HealthSmart was completed in September 2007. Preliminary evaluation reports have been developed for individual services and the entire Northern Alliance HARP-CDM program. An example using the AQoL data is provided below.

The Northern Alliance HARP-CDM program admitted 3458 patients to its services in the 12 months from September 2007. Analysis of the baseline quality of life data reveal that the mean utility score for our clients is 0.56 (n = 862, 95% confidence interval 0.54–0.58). This mean utility score equates to poor to fair self-reported health status and is statistically significantly worse (Fig. 1) than the Australian population data (mean utility = 0.83, 95% confidence interval 0.82–0.84).³

The Northern Clinical Research Centre has implemented a unique, highly coordinated approach to data collection, storage and reporting across 13 diverse services, and four health care organisations. All 13 services actively collect the dataset, which has been integrated with hospital utilisation and financial data, and stored within HealthSmart. As described above, these data will be regularly interrogated to generate routine reports which will facilitate benchmarking, reporting and governance.

These routine reports have immediate utility for the health service, however, this evaluation approach can also be viewed as an ‘enabler’ of future research. It is anticipated that as the HARP evaluation dataset grows, the evaluators will be able to address more detailed research questions of the HARP-CDM program. These questions may include:

• Which clients are benefiting most and least from HARP-CDM?

• Which services are the most and least effective? This question cannot be answered with usual evaluation processes.

• Which components of the models are the most and least effective in improving client outcomes, reducing hospital utilisation?

• What are the risk factors for re-presenting to hospital? Can these patients be identified early and how are they best managed?

A frequently voiced concern regarding disease management as a therapeutic strategy is that it risks treating the ‘worried well’, rather than people with the greatest health need. The evaluation has demonstrated that, the mean utility score of HARP-CDM clients equates to poor or fair self-reported health status.³ To the best of our knowledge this is the first time in Victoria that chronic disease services have objectively demonstrated appropriate targeting.

Health program evaluation has been defined by the World Health Organisation’s European Working Group on Health Promotion Evaluation as ‘the systematic examination and assessment of the features of an initiative and its effects, in order to produce information that can be used by those who have an interest in its improvement or effectiveness’.⁴ Common to all models and theories of good health program planning and evaluation is the integration of the evaluation plan with the program plan.^4–8 Theory suggests that by establishing the objectives of a program in the planning phase and clearly identifying indicators of success, an appropriate data collection strategy can be chosen and commenced with the implementation of the program. However, it may be argued that this approach can be inefficient, is resource intensive and that it is more suited to evaluating programs with a finite start and end date.

The evaluation strategy described in this paper was developed around the available administrative data rather than only being a priori hypothesis driven. This is seemingly a ‘back to front’ way of developing an evaluation plan; with the literature suggesting that one should start with the questions and develop a data collection and analysis strategy to best address these questions.^5,6 In contrast, this evaluation strategy started with the routinely collected available data, ascertained what questions it could not answer, then added essential missing data and developed possible analyses.

The evaluation strategy described in this paper is limited in it’s scope and in terms of external validity by the methods used to develop the approach. However, the methods used were a pragmatic approach driven by the limitations imposed by bringing evaluation to a large, complex program where no formal budget allocation was made available by the Department of Human Services for ongoing evaluation. In particular, it was considered vital to respect the imposition that evaluation can make on clinician time. Busy clinical staff collect the outcome data and thus the burden was limited by minimising the amount of data collection and by ensuring that all data items are carefully selected and fully analysed. Clinical staff are (appropriately) focussed on clinical service delivery and the experience of the evaluation team was that if outcome data collection was not efficient, then it would not be routinely collected by clinicians, especially during periods of increased clinical demand.

The evaluators are working with the clinicians to maximise data reliability and anticipate that many prospective assessments will be performed in the future including; client outcome evaluation, service model comparisons, and cost-utility analyses. The outlined approach has demonstrated that a highly coordinated ‘whole of system’ evaluation, which may contribute to the development of evidence-based policy, is feasible.