Impact and outcome evaluation of HealthPathways: a scoping review of published methodologies

KEYwords: HealthPathways; evaluation; care pathways.

All health systems aim to integrate care provision across the boundaries of primary, community, hospital, and social care. This ensures people receive a continuum of care including health promotion, primary prevention, disease treatment and rehabilitation within the health system in accordance with their needs. Successful health outcomes for patients depend on ensuring they receive the best, evidence-based, appropriate care. However, there is evidence from Australia that appropriate care is received in only 57% of health encounters, and that local clinical guidelines may have an important role in increasing the number of patients receiving appropriate care.¹ Even so, health-care providers’ access and use of available clinical guidelines have, to date, been unsatisfactory, due to factors such as duplication, differing recommendations, and inconsistent communication structure.² Incorporating local guidelines into online-based localised clinical decision support tools has the potential to improve patient care by improving the appropriateness of referrals, decreasing wait times and improving pre-referral investigation and treatment.^3,4 This in turn may improve patient outcomes, improve quality of life, and produce savings for health systems, resulting in high-value care.

The HealthPathways programme is an online health information system used mainly in primary health care that ensures a consistent and integrated approach to patient care within a local health region. HealthPathways originated in Canterbury, New Zealand, in 2008, and have since been adopted by various health systems in New Zealand, Australia and the UK.⁵ Currently, online resources called ‘pathways’ have been developed for over 550 disease conditions. These ‘pathways’ are localised to particular health regions by multidisciplinary teams of health professionals before their operationalisation. This ensures consistency of care according to the guidelines and standards agreed by local experts. Pathways are used in primary health-care settings as support during patient assessment and care, as well as providing clear referral pathways to secondary care services and alternative models of care available within each region. There has been a steady increase in both local health regions using HealthPathways and the number of primary health-care physicians accessing HealthPathways since its inception.⁶

After a programme is implemented, it is important for periodic evaluations to be conducted to assess its effectiveness. Programme evaluation involves systematically collecting and analysing data to assess the effectiveness and efficiency of the programme.⁷ Several attempts have been made to evaluate the HealthPathways at various stages after implementation.^8,9 A three-staged process has been proposed by several evaluators, which includes assessment of the implementation’s evaluation, impact evaluation, and outcome evaluation (Fig. 1).^8,10 Implementation (process) evaluation considers the effectiveness of implementation with respect to the pathway’s outputs, whereas impact evaluation assesses the use, behaviour and perceptions of HealthPathways by primary health-care professionals. Outcome evaluation assesses the effectiveness of HealthPathways in enhancing consistent care or treatment of health conditions and improving patients’ journeys.

Fig. 1.  Three staged process of HealthPathways evaluation.

The current lack of publications reviewing HealthPathways evaluation methodologies and synthesising evidence suggests that a scoping review would be useful to systematically map the work done in this area and identify gaps in knowledge and methodology. The aim of this study, therefore, was to perform a scoping review of the methodologies used in impact and outcomes evaluations of HealthPathways programmes. This review develops an evidence base for effective methodology to perform impact and outcome evaluations of these programmes in the future.

This review was performed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for scoping reviews.¹¹

This review was restricted to impact and outcome evaluations of HealthPathways and included only original research papers and published reports. Implementation evaluations and journal articles not presenting original research (opinions, editorials, protocols conference abstracts, and points of view) were excluded from the review. Research reporting the evaluation of HealthPathways as a secondary outcome were also excluded. To capture the full range of existing methodologies, evaluations could be quantitative, qualitative or mixed-methods in nature. No restrictions were placed on publication language, but we considered only records published after the inception of HealthPathways in 2008.

MEDLINE, Embase, CINAHL and Web of Science databases were searched using the keywords ‘HealthPathways’ OR ‘HealthPathway’. We searched for research published before 31 October 2020. We expected that some evaluations of HealthPathways may not be published in traditional academic sources, so we also searched grey literature for records, using a snowball method.

Titles, abstracts, and full-text of all potentially eligible records from the search were independently screened by two co-authors against the inclusion and exclusion criteria.

The lead author (SS) extracted all data for this review into an Excel spreadsheet (Microsoft Corporation). Year of publication, study population, study setting, study design, data collection (prospective, retrospective, or both), data source, programme aims evaluated, indicators, analytical method, and results were extracted for each included publication or report.

Based on the stages outlined in Figure 1, six programme aims were identified:¹⁰ (1) increased awareness and use of HealthPathways; (2) general practitioners are supported to adopt best practice, patient-centred care; (3) increased appropriate use of resources and services; (4) improved quality of referrals; (5) enhanced consistent care and management of health conditions; and (6) improved patient journey through the local health system. The study team decided to add ‘reduction in health-care cost and increase value for money’ as a long-term outcome, resulting in seven programme aims. We grouped publications by each aim to investigate the methods used to evaluate each.

A total of 139 articles and reports were identified from the initial search, including 66 duplicate records. Twenty-one studies were identified for the final review; 15 were research papers and six were reports (Fig. 2).

Fig. 2.  Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram for the scoping literature search performed.

Characteristics of the 21 articles and reports included in the review are shown in Table 1. Although HealthPathways were initiated in 2008, the first recorded evaluation was in 2013. Since then, an increasing number of evaluations has been conducted, with the highest number reported in 2018. A large proportion of studies were conducted in Australia (52.4%, 11/21). More than half the studies (57.1%, 12/21) used multiple study settings (eg general practice centres and hospital). General practice centres and hospitals were each used as settings in an almost equal number of studies.

Table 1.  Characteristics of the studies included in the review

Table 2 shows the number of studies evaluating each programme aim. ‘Increased awareness and use of HealthPathways’ was the most frequent programme aim evaluated (n = 12), followed by an equal number of studies (n = 8) evaluating ‘improved quality of referrals’, ‘enhanced consistent care and management of health conditions’ and ‘improved patient journey through local health system’. One study evaluated the economic impact of HealthPathways⁹ and no identified research had assessed ‘increased appropriate use of resources and services’.

Table 2.  Number of studies evaluating each programme aim

All included evaluations and their aims, indicators, settings, study designs, populations, data sources, data collection, and analysis methods are provided in Table 3.

Table 3.  Different methodologies used to evaluate the programme aims

Of the 12 studies evaluating awareness of use of HealthPathways, one compared pre- and post-data to evaluate the significance of changes over time,² and one other study used a mixed-methods approach (qualitative and quantitative methods).¹² Google analytics was the most common data source (n = 9) to analyse usage patterns. The most common indicators used in Google analytics were number of users per month, number of pages viewed per session, number of views per month for each page and clinical stream, and ‘often viewed’ pages. Apart from Google analytics, prospective data collection methods such as online surveys (n = 4) and telephone surveys (n = 2) were used to evaluate the awareness and use of HealthPathways. Only five studies used a combination of retrospective (eg Google analytics) and prospective data sources (e.g. online surveys and telephone surveys) to describe a wider scope of activity. Participants in the studies were mainly general practitioners (GPs) (n = 8), with the remainder comprising a balance of other stakeholders, including hospital staff and practice managers.

This programme aim focused mostly on evaluation of barriers and facilitators to HealthPathways use. Of the five studies evaluating this aim, four used qualitative methodologies to identify barriers and facilitators. Qualitative inquiry was carried out among GPs, the members of the HealthPathways workgroups and hospital staff, through face-to-face interviews and focus group discussions. Apart from identifying barriers and facilitators, other important indicators used to evaluate this aim were: HealthPathways as a learning platform for GPs (assessed using a qualitative study) and improvement in the capacity of GPs (indicated by a reduction in referral rates).

The quality and completeness of referrals are two indicators used to evaluate this aim. Evaluators of the Hunter and New England HealthPathways used an audit tool adopted from the Royal Australian College of General Practitioners (RACGP) Standards for general practices (4th edition) referral documents, to demonstrate change in quality and completeness of referrals after implementing HealthPathways.^2,13 In five studies (out of eight), pre-post study design was used to demonstrate improvement in the quality and completeness of referrals. These studies used hospital records to extract data from a random sample of pre-implementation and post-implementation referrals (eg 50 referrals each). They used referrals from several selected patient groups (eg patients attending antenatal care, radiology service, haematology clinic) rather than referrals from a general patient population. Other indicators used by different studies were appropriateness of the referrals, acceptance rate of referrals from primary health-care settings, and proportion of urgent referrals requested.

Of the eight studies evaluating this aim, four were conducted among specific patient populations, such as patients with acute pancreatitis, post-menopausal bleeding, cardiovascular disease, diabetes, urological disease, and respiratory disease. Qualitative inquiries (focus group discussions and face-to-face interviews) were conducted with GPs, members of the HealthPathways workgroup, and hospital staff to assess their subjective perceptions of the impact of HealthPathways on patient care.

With the exception of one report addressing this aim,¹⁴ all studies assessed the wait time for specific patient groups (eg patients with diabetes mellitus, hospital referrals in antenatal care, persistent non-cancer pain, and suicide risk). Retrospective analysis of hospital records was the most frequent method of measuring this outcome (n = 6/8). In addition, Chow et al. used a patient survey to assess wait time for an appointment at their diabetes clinic.¹⁵ Apart from wait time, Huckel Schneider used qualitative methods to measure acceptability of HealthPathways among patients as an indicator of improved patient journeys.¹⁴

Only one study evaluated this programme aim. Blythe et al. estimated the annual cost savings of inappropriate specialist visit avoidance per patient among patients admitted or referred from primary care for cardiovascular disease, diabetes, urological disease, and respiratory disease.⁹

This scoping review assessed 21 published HealthPathways evaluations, describing the methodologies used to determine impact and outcomes across a variety of programmes. The impact and outcomes framework for this scoping review was developed using the six aims identified in a South Australian HealthPathways report,¹⁰ with one additional aim identified during the process of this review. These seven aims included components related to awareness, usage, improvement in patient care and patient experience, and providing best value for money. Much of the previous focus in the evaluation literature has been on measuring increased awareness and use of HealthPathways, but several evaluations have also assessed improvements in care consistency, better care of health conditions, improved patient journey, and the impact on GPs. A wide variety of quantitative, qualitative, and mixed methodologies have been used for evaluation, and overlap has occurred in many data sources and analytic methods. There are benefits and limitations to each of these evaluation approaches.

Google analytics data, a service provided by Google that tracks and reports website traffic, was the most common data source used to analyse usage patterns. The use of HealthPathways is best described by tracking individual users over time. However, because Google analytics track users by using cookies, it limits tracking of individuals as it is specific to devices and browsers. Therefore, the number of unique users could be an overestimation, whereas the number of return users is likely an underestimation.¹⁶ This is a well-documented limitation of Google analytics data. An alternative data source to overcome this limitation is individual user surveys. Most user surveys included in the review had used online surveys, which also have limitations. Poor response to online surveys is common among GPs, potentially introducing non-response bias, and could lead to erroneous conclusions. Furthermore, GPs who are more engaged with HealthPathways could respond more often to these surveys, introducing selection bias.¹⁷ Mixed-method approaches using quantitative methods such as Google analytics and user surveys, complimented with qualitative methods, could be the ideal way of evaluating the aim ‘increased awareness and use of HealthPathways’ while addressing some of these limitations.

Qualitative research methods are also best placed to gather the in-depth insights required to understand the barriers and facilitators to HealthPathways use, as this information is currently not well understood by other means.¹⁸ This was demonstrated by the fact that most of the evaluations in this review applied this methodology to describe context-specific HealthPathways barriers and facilitators. However, data collection and analysis of qualitative studies requires expertise and can be time-consuming, and findings may not be transferable to other contexts. Furthermore, the perspectives of all relevant stakeholders may not be adequately captured during qualitative inquiries.

Evaluations of the aims ‘improved quality of referrals’, ‘enhanced consistent care and management of health conditions’ and ‘improved patient journey through local health system’ were conducted among specific patient groups and specific HealthPathways, such as patients with acute pancreatitis, post-menopausal bleeding, cardiovascular disease, diabetes, urological disease, and respiratory disease. Attention must be given in selecting disease conditions or HealthPathways, as inappropriate selections could result in a skewed representation of the evaluated aim. The Hunter and New England HealthPathways evaluation team used routine antenatal care, persistent non-cancer pain and suicide risk based on the criteria that HealthPathways had been complete and in operation for ≥12 months; the HealthPathways addressed conditions of high prevalence in the community and had highest utilisation among all Hunter and New England HealthPathways; there were no major changes to the pathway in the previous 6 months; and they had been developed according to the Hunter and New England HealthPathways model. Although the quality and completeness of the referrals has been evaluated, the accuracy of the information in the referrals has not been evaluated in any of the reviews, which may be due to the difficulties in the practicality of collecting these data.

Blythe et al. assessed the cost implications of HealthPathways by estimating the cost saving due to avoiding inappropriate specialist referrals.⁹ This is the only evaluation we found that assessed the cost implications of HealthPathways; however, the authors had made many assumptions in calculating cost savings, signalling the challenges in conducting such a study. None of the studies had conducted a standard cost-effectiveness analysis to evaluate whether HealthPathways produce best value for money. A high-quality cost-effectiveness analysis would require data linkage between primary care and hospital admissions using patient-level data. In countries like Australia, economic analyses should include patient-level general practice and hospital utilisation data, patient-level clinical data (at both general practice and hospital level), and individual GPs’ adherence to HealthPathways, all of which are not readily accessible to researchers, even after necessary ethics approvals. However, in countries like Australia, appropriate data linkages with necessary administrative willingness, policy changes and changes to privacy laws could improve the possibility of comprehensive evaluations in the future.

This review describes the strengths, limitations, and gaps of the current HealthPathways evaluation literature. Although some impact and outcome aims have been evaluated extensively, aims such as increased appropriate use of resources and services, reduction in health-care cost and increased value for money have rarely been evaluated for HealthPathways. Assessing the impact or the outcome of HealthPathways independent of all other interconnected factors is challenging and may not be achievable. However, comprehensive HealthPathways evaluation should attempt to evaluate as many of the aims presented in this review as possible. Each aim may require specific methodologies sensitive enough to capture the impact the HealthPathways are making over time in a specific area.

The authors declare no competing interests.

This study was funded by HealthPathways South Australia. Health Pathways South Australia is a tripartite partnership between Wellbeing South Australia, Country South Australia Primary Healthcare Network (PHN) and Adelaide PHN. The funders had no role in the design and conduct of the study, data collection, analysis or in article preparation and approval.

The data that support this study are available in the article.