Predictive risk modelling in health: options for New Zealand and Australia

In health care systems across the world, a small fraction of the population accounts for a disproportionately large proportion of health care usage and cost. For example, in the United States, Reuben et al. estimate that 10% of all Medicaid beneficiaries account for around 70% of all expenditure.¹ Likewise, Hughes et al. found that in 1998, Medicare’s sickest 15.3% of patients accounted for 75.7% of expenditure.² In the United Kingdom, the Department of Health reports that around 5% of patients account for 40% of inpatient bed-days.³ South Auckland Health⁴ reports that in 2001, 14% of patients accounted for 44% of adult medical bed-days (national data are not available in New Zealand). Since resource use and costs are so unequally distributed in the population, this raises important questions about how health systems should provide care and manage the costs of those people who are the most intense users of resources.

The people who use most health care resources tend to be older people and those with chronic conditions. The World Health Organization describes care for chronic conditions as being ‘the health care challenge of this century’, with such conditions currently responsible for 60% of the global burden of disease.⁵ Hutt et al. found that a large proportion of emergency hospitalisations are for people with chronic conditions,⁶ and figures published by the UK Department of Health indicate that two-thirds of patients admitted as a medical emergency have a long-term condition.³ Therefore, as people live longer with increasingly complex conditions, health care systems must adapt in a sustainable way.

The Atlas of Avoidable Hospitalisations in Australia: Ambulatory Care-sensitive Conditions shows that in Australia there was a total of 552 000 avoidable hospitalisations in 2001–02. This was concentrated among people with chronic diseases. Of the 680 337 emergency presentations to Victorian Emergency Departments in 2000–01, there were 83 542 re-presentations from 15 440 patients who attended on at least four occasions and who accounted for 12% of all ED presentations.⁷

In the New Zealand context, the aging babyboomer population is currently reaching retirement age. Statistics New Zealand estimates that by 2051, the number of people aged 65 and over will have more than doubled in size compared with 2004, and the numbers aged over 85 years is set to increase more than five-fold during this time.⁸ Older people often have multiple chronic conditions: for example, in the Ministry of Health’s 2002 health survey of those over 65 years, 20% of females and 15% of males had four or more chronic conditions.⁹ Therefore, New Zealand faces a difficult challenge in coping with the sharply aging population and the projected rise in the prevalence of chronic conditions. The Ministry of Health predicts that by 2051 older people’s share of health expenditure will have increased from 40 to 63%, compared with 2002.¹⁰

The literature refers to the small percentage of people who utilise a disproportionate amount of resources as being ‘high risk’. A literature review by the King’s Fund notes that there are two principal reasons for seeking to identify these ‘high risk’ individuals who are likely to be the most costly.¹¹ The first reason is to determine appropriate insurance premiums or resource allocations to cover their likely future health care costs. When used within insurance-based health systems, this process is referred to as ‘risk adjustment’. Given the skewed nature of health care costs, insurers who predict the high-cost patients, and price their premiums appropriately, will be more profitable than those that charge a uniform premium. Risk adjustment has therefore been a central pillar of profitability in insurance-based systems and has become highly sophisticated in the United States. In the United Kingdom, the Nuffield Trust has developed a Person-Based Resource Allocation (PBRA) formula that uses highly detailed epidemiological data to set the income of GP practices based on the anticipated costs of individual registered patients.¹²

The second reason for identifying high risk patients, which may be of more interest in Australia and New Zealand, is to identify patients so that they can be offered a preventive intervention. This is known as ‘Case Finding’ and is aimed at simultaneously improving care and reducing net expenditure. In these circumstances, the predictive tool is used to target ‘upstream’ interventions at those patients who are deemed most likely to incur ‘downstream’ costs of emergency hospitalisation. In theory, economic benefits will result both directly (from any savings that accrue if the intervention costs are lower than the cost of treating the hospital episode) and indirectly (from a more efficient allocation of resources, such as the opportunity for better workload management due to fewer medical emergencies). However, any success is highly dependent on the design of the intervention.

Studies evaluating case finding for preventive interventions have, to date, largely concentrated on the impact on avoidable hospital usage. For example, Parker, McCombs and Grady studied the risk of unplanned hospital readmissions¹³; McCusker, Bellavance and Belzile focussed on early and frequent return to the Emergency Department¹⁴; and Wahls, Barnett and Rosenthal looked at primary care and outpatient visits.¹⁵ Other risks, where costs might be less readily quantified, include death, functional decline,¹⁶ clinical complications¹⁷ and other adverse health events.¹⁸ Although death is not necessarily a high-cost event in itself, the last year of life typically involves high health care costs and is therefore of great significance to health providers and insurers.¹⁹

The King’s Fund literature review outlined three main methods for identifying high risk patients: ‘threshold modelling’, ‘clinical knowledge’ and ‘predictive modelling’.¹¹ The first approach, threshold modelling, is also referred to as ‘rules-based’ or ‘criterion-based’ modelling. It uses a set of predetermined criteria to determine which patients are at high risk of an adverse event. The method identifies all patients that meet a specified threshold for the parameter of interest. For example, a threshold model identifying patients at high risk of emergency readmission might use the threshold of anybody who is aged 65 years or over and has had five or more admissions in the previous 12 months. Such models were previously used in the UK for case-finding projects; for example, at the Castlefields Health Centre.²⁰ However, the evidence suggests that threshold models do not yield a high degree of accuracy within a general population, although they have proved to be more accurate when used within a specific clinical context, such as identifying those at risk of coronary heart disease.³ An example of the use of a threshold model is the Frequent Adult Medical Admissions Scheme (FAMA) used by the Counties Manukau District Health Board in New Zealand. This scheme is designed to reduce patient bed-days by providing high risk patients with intense community based care plans. The service was offered to those who had more than two medical admissions lasting over 5 days each within a 12-month period.²¹

In general, threshold models are predisposed to the negative effects of selection bias and regression to the mean. Selection bias occurs when individuals are selected because they are outliers who represent an extreme. This means the model suffers from the problem of regression to the mean – a situation where those people who are extreme one year (e.g. in terms of number of hospital admissions) are rarely extreme the next year. For instance, the costs of patients who are ‘expensive’ in one 12-month period are likely to be lower in the subsequent year even without intervention.²² The reasons for this phenomenon include self-limiting illnesses, definitive treatment, change in social circumstances and death. Because of regression to the mean, threshold models tend not to identify those individuals who are at highest risk in the subsequent period. For instance, Weissert, Chernew and Hirth found that 75 to 100% of those judged by a threshold model to be at risk of immediate or near-term nursing home entry in fact stayed out of a nursing home for an entire year or longer without any intervention at all.²³ They concluded that there is a large misallocation of Medicare funded home health care due to the inaccuracies of threshold models.

Roseman’s evaluation of the FAMA project in Counties Manukau found that patients who received intervention did experience a decline in bed-day usage.²¹ However, although this decline was dramatic, bed usage in the year following intervention was similar to the bed usage of these same patients in the 2 years before intervention. This suggests that the threshold model might have been identifying patients whose course of illness was naturally waning – and that what appeared to be a reduction in bed-days following FAMA may have occurred due to a regression to the mean.

The second approach to identifying high risk patients relies on using clinical judgment. Here the clinician uses his or her knowledge, training and judgment to identify individuals who are thought likely to require emergency hospitalisation and would benefit from an intervention. This approach is widely used in many countries where GPs, social workers, and other health and social care professionals refer patients to specific interventions based on their belief that these individuals would benefit from those interventions.

Little formal evaluation has been conducted to assess the relative accuracy of clinical knowledge in predicting future risk, but the existing evidence suggests that this approach has a low level of predictive accuracy. Clinicians may be able to identify patients who are currently high risk, but are less able to identify those who are going to be at high risk in the future.²⁴ One study that examined the accuracy of clinicians in predicting the readmissions of patients with schizophrenia indicated that fewer than 20% of readmissions were predicted, and even this was only achieved because of the relatively homogenous nature of the local population with mental illness.²⁵ Moreover, clinician assessment is limited to those patients who already have contact with a service. Therefore, using clinicians to predict risk across a large, enrolled population is relatively inefficient and inaccurate.

A third method for identifying high risk patients is to use a predictive risk model (PRM). This is a statistical model specifically designed to provide early warnings about a patient’s risk of an adverse episode in the future. PRMs seek to establish a statistical relationship between a set of routinely gathered predictor variables and the occurrence of an adverse episode in a future period. These relationships can then be used to forecast the likelihood that any given patient will have an adverse episode in the future based on that patient’s predictor variables. PRMs assign a distribution of future risks over an entire population, and because they are based on routinely gathered data (such as demographic and utilisation history), they can ‘risk stratify’ the entire population. Unlike the other two methods for case finding, which focus on identifying the highest risk patients, PRMs are able to identify patients at all levels of risk. As there are far more people at intermediate risk, the potential impact could be substantial if less expensive, cost-effective interventions could be developed for these people. As well as identifying high risk patients more accurately than either threshold modelling or clinical judgment,²⁶ PRMs can also determine the risk of the whole population and do so on a frequent basis (typically monthly).

Another important aspect of a PRM is that it uses routinely collected data (e.g. demographics, utilisation) which are captured for administrative purposes (e.g. for billing or quality monitoring). Therefore, implementing a PRM may only require minor adjustments to existing data infrastructures.

The general concept of predicting patient risk is not new to the Australian and New Zealand health systems. For example, the use of desktop or computerised tools for predicting cardiovascular risk is well-established for targeting treatment in the General Practice setting.²⁷ However, such risk prediction tools are typically designed to support clinicians in making treatment decisions. As such, they tend to be disease-specific. By contrast, the PRMs described in this article focus on predicting more general adverse events, such as the use of a specific service (e.g. emergency hospitalisation). The PRMs of the sort we are considering here are intended to influence the design of the whole system of care. They may be regarded as the health service equivalent of disease risk assessments.

The NHS in England provides one of the first examples of a national health care system using a PRM to identify patients for the purpose of targeting preventive interventions. One of the motivations for developing PRMs in England was an article by Feachem and colleagues, which argued that Kaiser Permanente (a managed care organisation based in California), was providing higher quality health care than the NHS and at a lower cost.²⁸ One explanation for Kaiser Permanente’s success was its use of PRM to identify patients at high risk of hospitalisation and then to manage these patients intensively, with the aim of reducing emergency hospitalisation rates. In 2004, the UK Department of Health published a report which found that a small percentage of the population spent a disproportionately large amount of time in the hospital and used a disproportionately large amount of resources.³ It also announced its objective of reducing ‘emergency bed-days by 5% by 2008 [compared with 2003–04 baseline] through improved care in primary care and community settings for people with long-term conditions’.²⁹

In response, Essex Strategic Health Authority (on behalf of the 28 Strategic Health Authorities), the UK Department of Health and the NHS Modernisation Agency commissioned the King’s Fund, Health Dialog, Inc. and New York University to undertake a Predictive Risk Project. This project produced a literature review¹¹ and two PRMs: ‘PARR’ and the ‘Combined Model’ (see Box 1).

Box 1.  PARR and combined model

The PARR (Patients-At-Risk-of-Rehospitalisation) tool predicts a patient’s risk of rehospitalisation in the coming 12 months. It uses hospital episode statistics and variables from the Census. The first version of PARR was launched in the autumn of 2005. According to Billings et al., out of the 10% of the riskiest patients according to PARR, 90% were rehospitalised within a year.³⁰ The model’s developers published a chart listing combinations of the patient risk segment, the success rate of the preventive intervention to be offered, and the per-patient intervention cost. This chart aids the design of cost-effective interventions. PARR has been released in several versions. PARR1 restricts the analysis to patients who have had a hospital admission for an ambulatory care-sensitive condition, i.e. a condition in which ‘…timely and effective outpatient care can help to reduce the risks of hospitalisation’³¹, whereas PARR2 covers all conditions. PARR+ and PARR++ relate to the software interface through which users run the model.

A second PRM was developed later as part of the same project. Called the Combined Predictive Model, it identifies a patient’s risk of hospitalisation (rather than rehospitalisation) in the coming 12 months. In contrast to PARR, the Combined Predictive Model includes the entire registered population and not simply those who have had a recent hospitalisation. The Combined Predictive Model uses a more comprehensive dataset of outpatient data, accident and emergency data, and general practice data from electronic medical records in addition to inpatient and census data. This model was launched in December 2006.

In Wales, PRISM (Predictive Risk Stratification Model) is analogous to the Combined Predictive Model, although it is run centrally through the Welsh Predictive Risk Service (a secure website) rather than being run locally, as in England.³² In Scotland, the SPARRA (Scottish Patients At Risk of Readmission and Admission) model is in use, which has some similarities to the PARR model in England.³³

PRMs can vary in four principal ways: (1) the event they are predicting; (2) the set of patient predictor variables they use; (3) the time period over which they predict risk; and (4) the type of statistical technique they use.

Evidence indicates that the accuracy of a PRM depends largely upon which patient predictor variables are used.¹¹ The possible patient predictor variables can be loosely categorised into the following groups: socio-demographic; diagnostic; prior utilisation or costs; pharmacy data; health status and functionality; and clinical data. The literature on which predictor variables produce the most accurate PRM is large, contradictory, and dependent on the event being predicted. Part of the contradictory conclusions may result from the fact that measures of accuracy differ between articles. Many authors agree that demographic variables alone do not yield high predictive power and that diagnostic and prior utilisation variables significantly increase predictive power when added to demographic variables. The inclusion of pharmacy data has been shown to add power in a small number of studies, as has the addition of health status and functionality information. Very few studies have included clinical and procedural information, and the marginal improvement in accuracy from including these variables has not been established.

PRMs can also have different time frames over which they predict the risk of an event. The length of this period has been shown to have a major impact upon the predictive accuracy of PRMs. There is consensus in the literature that a PRM with a time period of less than a year produces more accurate predictions than a PRM with a time period of greater than 1 year.^2,24 Mukamel has found that predictive ability declines as the predictive period increases.³⁴ Thus, the predictive ability would be higher for the following year than for a later year. Hughes analysed the same model for predicting costs based on diagnostic data, but with two different time frames. The model with a time period of less than 1 year produced an R² value of 42.75, whereas the model with a time period of greater than 1 year produced an R² value of 10.66². Unfortunately, PRMs with a time period of shorter than 1 year may be less useful for both risk adjustment and for identifying high risk patients for an intervention, because of the lag times in acquiring data and because high risk patients need to be identified with sufficient time for the intervention to make a difference.

Finally, PRMs can be built using a variety of statistical techniques. The literature on this subject is extensive, yet it is clear that there is no consensus as to which technique is best. Many predictive models, including those used by the NHS in England, Scotland and Wales, are based on regression. This is a technique used to assess the relationship between the patient predictor variables and a dependent variable representing the adverse episode. The relationship may be non-linear, but it requires the modeller to specify the exact functional form of this relationship.

PRMs may use linear or logistic regression techniques, or both. The main difference is that linear regression is more suitable when the outcome variable is continuous whereas logistic regressions are used when the outcome is binary. Binary variables have two categories that represent an event or characteristic of interest (e.g. whether an individual is ‘high risk’ or not). However, both types of model can be used to rank individuals in order of predicted risk. These predictions can be used to target preventive care to those patients with the highest risk or those whose predicted risk is above a predetermined cutoff point. There is no consensus in the literature about whether linear or logistic regressions are more accurate. Zhao et al. and Ash et al. favour multiple linear regression models^35–38; Schatz et al. and Roblin et al. favour multiple logistic regression models^39,40; whereas Meenan et al. and Dove, Duncan and Robb use both multiple linear and logistic regression models.^41,42 Meenan and colleagues found that a linear regression model was superior to a logistic version of the same model. However, they noted that both models performed similarly when using a policy-relevant threshold.

In recent years, some developers of PRMs have used statistical techniques based on artificial intelligence (AI). Such PRMs are widely used in the financial, legal and actuarial sectors and by certain health care companies in the US for disease management. The advantage of this technique is that the functional form does not have to be specified by the modeller. Model developers use a wide range of techniques, including neural networks, regression (linear, polynomial or logistic), decision trees, fuzzy logic, principal component analysis, rule induction, genetic algorithms and Kohonen networks.⁴³ However, the most commonly used AI models are based on neural networks. Evidence suggests neural network models can yield higher predictive power than typical regression models. One study found predictive power to be double that of a traditional regression model.⁴³ Despite the evidence of improved accuracy, the adoption of neural network PRMs has been slow within health care. This may be due in part to a concern among clinicians that such models represent a ‘black box’ where it is difficult or impossible to determine how the neural network associates patient predictor variables with the risk of an adverse episode.

There are several ethical issues regarding the use of PRMs for stratifying populations and allocating resources. One issue is the sensitive nature of health care data and the need to maintain privacy. In most jurisdictions, there exist fairly well-defined rules that protect patients and restrict the way in which providers may deploy health data (e.g. Health Insurance Portability and Accountability Act (HIPAA) in the US; Privacy Commissioner in New Zealand; and the National Information Governance Board for Health and Social Care (NIGB) in the United Kingdom). For example, in England the PARR model and the Combined Predictive Model are run on pseudonymous data, where all identifiable variables (name, date of birth and address) have been removed, and the unique key (NHS number) has been scrambled. This renders the data effectively anonymous until the patient’s own GP unscrambles the pseudonymous NHS number to identify the patient’s predicted risk score.

A more difficult ethical issue is that, when utilising a PRM, both needs-based indicators (e.g. diagnosis) and non-needs-based indicators (e.g. sex) are used to predict risk and to allocate interventions. Kass argues that there is an ethical imperative to implement programs in ways that ensure that benefits are not targeted solely to one group.⁴⁴ For example, cardiac prevention programs should not be targeted solely to men simply because they have a higher average risk of a myocardial infarction than females. Such a policy ‘unfairly’ denies access to women, some of whom might upon closer examination be found to have a higher risk than some males. Although PRMs that use multiple predictor variables can produce much subtler results, any predictor variable based on non-clinical data (such as age, sex, ethnicity or deprivation) needs to be justified carefully, because issues of fair and equitable resource allocation may arise.

In the New Zealand or Australian context, we would suggest that a PRM might best be regarded as a planning and case finding tool for the health service. In many cases it may be cost-ineffective to offer intervention packages to everyone, but be highly cost-effective if intervention packages are targeted to patients based on their risk profile. In other words, a PRM ought to be viewed as a tool for resource management instead of a clinical tool to predict specific diseases.

In New Zealand, the National Minimum Data Set (NMDS) provides ideal data to develop a National PRM. The NMDS is a national collection of public (and some limited private) hospital discharge information, including clinical information for inpatients and day patients. All records have a valid unique patient identification number. Data have been submitted electronically in an agreed format by public hospitals since 1993 (in earlier years, the data were of limited quality). The NMDS offers routinely collected, patient-level data and a set of potentially useful predictor variables which could be used to build a New Zealand equivalent of a PARR model. In the future it might also be possible to link pharmacy and GP data with the hospital discharge data in order to build an equivalent of the Combined Predictive Model. Similarly, in Australia the National Minimum Data Set for Admitted Patient Care, which collects detailed information on the separations of all emergency admissions across the private and public acute sectors, could be used. The Person Identifier can be used to link hospital admissions longitudinally which could then be used to build a PRM to predict emergency hospitalisation.⁴⁵

An alternative to building new models using the NMDS would be to purchase a commercial PRM from an established vendor (which includes 3M, ACG, Verisk, D2Hawkeye, Health Dialog, Ingenix and MedAI) or seeking permission to adapt an existing open-source model (e.g. PARR). We believe that a customised New Zealand or Australian model would be preferable because commercial PRMs developed in other countries are based on different patterns of demography, social circumstances, epidemiology and health care utilisation.

The next step in this project is to produce a ‘proof of concept’, where we would develop a PRM for a specific user (e.g. District Health Board or a large Australian insurer), test its usefulness and couple it with an intervention. There are many examples of large scale IT projects in health that over-promise and under-deliver. To avoid these pitfalls, we believe that small scale, grassroots applications that are close to the user may be a better approach for exploring the applicability of PRM in Australia and New Zealand than a large scale, top-down approach.

The authors declare that no conflicts of interest exist.