Acute admission risk stratification of New Zealand primary care patients using demographic, multimorbidity, service usage and modifiable variables

Keywords: algorithmic, care planning, Health Care Home, hospitalisation, morbidity, primary health care, risk assessment, risk stratification.

Risk stratification in primary care aims to rank patients according to their chance of experiencing an upcoming adverse health outcome, then partition the patients into risk tiers. Patients in different risk tiers can be prioritised for different methods of care, to try to prevent the manifestation of potentially avoidable adverse health outcomes, or the inappropriate use of secondary care services. In New Zealand’s Health Care Home model of patient-centred primary care, the primary use–case for a risk stratification tool is to help general practices (GPs) identify complex, high-risk patients, so that proactive care plans can be developed for these patients, and longer appointments can be scheduled.¹

Risk stratification can be done algorithmically or by using clinician judgement. Studies have been undertaken to determine whether clinician judgement outperforms algorithmic risk stratification models or vice versa, for predicting 30-day hospital readmissions, but their mixed conclusions suggest neither method unambiguously outperforms the other.^2–4 Algorithmic risk stratification, however, has the advantage of reducing clinician workload and will undoubtedly improve as datasets increase in size and quality.

Existing risk stratification tools in the literature include the PEONY and QAdmissions models,^5,6 both derived using data from the United Kingdom, and a New Zealand implementation of the PEONY model.⁷ These models have reasonable discriminatory capability, with areas under the receiver operating characteristic curve (AUROC) of 0.8, ~0.78 and 0.72 respectively.

Recent feedback from GPs was an existing, proprietary risk stratification tool, from which monthly updated predictions are provided to practices in Wellington, Kāpiti Coast and Wairarapa, was not being used successfully or on a regular basis. There was a general feeling that the highest risk patients are well-known to the practice so the use of algorithmic risk stratification does not add much value. Another concern raised is that only the highest-risk tier of patients is being identified for proactive care, whereas others in slightly lower risk tiers could, with appropriate interventions, avoid moving into the highest risk category altogether.⁸

This work aims to improve upon existing stratification tools, both in predictive performance and wider applicability, and to demonstrate the value of the tool to practices. Better assessing risk for young patients, as well as for those with mental health conditions, are also key objectives of this study. The resultant risk calculations are to be provided to GPs as a list of patients with >50% chance of an acute admission in the next 6 months.

To derive the acute admission risk stratification model, an open cohort is defined starting in 1 March 2017 and finishing in 1 November 2021, comprising patients registered with a GP in the Tū Ora Compass Health primary healthcare organisation (PHO). In the open cohort, patients enter the dataset on or after the March 2017 start date, provided they have been registered with a Tū Ora Compass Health practice for at least 12 months prior to entering the cohort, and there is a minimum of 6 months of follow up. Patients of all ages are considered, but given these registration constraints, only those aged >12 months can form part of the cohort. The latest date a patient can enter the cohort is April 2021. A patient leaves the cohort when:

• a continuous period of registration is broken, either through mortality or leaving the suite of practices;

• the patient has an acute admission, or;

• the end date of 1 November 2021 is reached.

Primary care data are available before this start date, from as early as June 2003, to inform previous patient diagnoses. Hospitalisation data are available from as early as January 2010, but only for the two study regions, the Capital & Coast and Wairarapa District Health Boards (DHB). For this reason, patients who reside outside the Capital & Coast or Wairarapa DHB catchments, even if they are enrolled in a practice inside the study area, are removed from consideration. As hospitalisation data are not available for other nearby hospitals, patients living outside the Capital & Coast and Wairarapa regions may utilise hospital services near their residence rather than near their enrolled practice, which may bias their hospitalisation risk as low. Patients with a missing National Health Index number are also removed from analysis.

For the modelled dependent variable, an acute admission is defined as an unplanned visit to hospital that requires >3 h of care excluding wait times. Arranged and elective admissions are excluded. Numerous covariates are used to model acute admission risk, relating to patient demographic, recent utilisation of services and morbidity (Supplementary Table S1). The datasets used to derive these covariates are the quarterly primary care registers for the practices under the Tū Ora Compass Health umbrella, the scripts prescribed by the practice and recorded in their patient management system (PMS), GP read codes inputted into the PMS, patient-level hospitalisation data from regional Capital & Coast and Wairarapa hospitals, and claims data from the practice to Tū Ora Compass Health.

Patient morbidity is often defined using various local or international indices (eg the Charlson Comorbidity Index (CCI),⁹ the Elixhauser Index,¹⁰ or many others).^11,12 These indices are derived by grouping different condition diagnoses, and summing the weighted contribution of each condition to form a final comorbidity score. The condition codes for each category are obtained from International Classification of Diseases (ICD)-10 hospital discharge codes,¹³ or translations for Read/OXMIS-coded databases.^14,15

Determining patient morbidity using GP coding may not capture the true morbidity, and in particular, the quality of coding varies by condition and the meticulousness of the practitioner.¹⁶ This study has access to both hospital, GP classifications and prescribing datasets, however, and by combining these datasets, patient comorbidity and its implications can be better quantified.¹⁷ Practice-specific parameters are also added to the statistical model, to mitigate any bias that may come from certain practices being less fastidious in their morbidity coding. The hospital discharge codes use the ICD-10 system, whereas GP classifications use a combination of Read codes and SNOMED (Systematized Nomenclature of Medicine) codes depending on the PMS. For this study, SNOMED codes were translated into Read codes using a PMS-provided table.

This study compares the ability of the CCI and Elixhauser Index comorbidity categories to predict acute admissions with an expanded set of categories in Supplementary Table S1 that provided more granularity. The codes and search terms to define the categories in Supplementary Table S1 are also included in Supplementary File S1.

Numerous additional covariates were tested for inclusion in the model, including ones that parameterise the patient demographic, GP characteristics, and various common diagnoses, but these were removed from the final model due to their very small effect size relative to the uncertainty.

To model patient acute admission risk, most studies typically adopt logistic regression or survival regression models such as the Cox proportional hazard model. This work uses a type of survival regression, the log-logistic accelerated failure time (AFT) model.¹⁸ The log-logistic distribution was selected to allow for both monotonic and non-monotonic hazard functions. The advantages of the AFT model are computational efficiency and the intuitive interpretation of coefficients.

Three different patient morbidity models are tested, which summarise patient health status in different ways:

1. the Charlson Comorbidity Index categories

2. the Elixhauser Index categories

3. the categories defined in Supplementary Table S1, and graphed in Fig. 1.

The predictive capabilities of the three alternative models are compared to assess whether the more granular categories in Supplementary Table S1 are justified by the data.

Fig. 1.  Accelerated failure time regression model coefficients related to patient morbidity, for a model estimating patient acute admission risk (2017–21). Points correspond to the mean coefficient value, and the horizontal lines correspond to two standard errors either side of the mean.

The data are randomly partitioned into training and holdout sets containing 80 and 20% of the data respectively. Model selection was performed by calculating Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC) and the area under the receiver operating characteristic (ROC) curve on the holdout set. Using the holdout set, the models were also compared for their false positive rate and true positive rate on ahigh-risk prediction threshold of 0.5, to assess which model better extracts the highest risk patients.

A total of 319 943 patients satisfy the criteria to enter the cohort, with 34 231 patients removed for living outside of the Capital & Coast and Wairarapa DHB catchments. The patients in the cohort have a total of 1 052 781 patient-years of follow up. And 61 170 patients had an acute admission during the study period, giving a crude rate of 5.8% per year. Patients with missing data were removed from the analysis, except those without an available deprivation index quintile value, who were assumed to have the highest deprivation (a quintile value of 5). Supplementary Table S2 details the cohort characteristics in terms of the model covariates, as well as each covariate’s rate of acute admission per year according to the dataset. Table 1 shows the test metrics for the three alternative models (A–C), as well as models that remove a key dataset (D–G) to determine the most critical datasets for assessing acute admission risk. The difference in predictive capability between the three ‘full models’ that define patient morbidity in different ways is very small, at least in terms of the area under the ROC curve. In particular, the models with the study-specific morbidity categories (A), and the Elixhauser Index morbidity categories (B) are very similar, with model A slightly preferred according to all test metrics but one. Model D, with no consideration of patient morbidity whatsoever, also has similar predictive capability to the models that do consider patient morbidity. The predictive performance drops markedly when removing the variables related to healthcare service utilisation, indicating that the patient risk factors are more strongly determined by demographic and recent healthcare service utilisation.

Table 1.  Test metrics for seven candidate accelerated failure time survival regression models, using different suites of predictor variables, fitted and tested using data from 2017 to 2021.

Given that models A and B have similar predictive capability, the choice of preferred model is largely a pragmatic one rather than data-driven. Model A is the preferred model in this study. The morbidity categories in Supplementary Table S1 have been carefully derived to account for read codes that are commonly used by local GPs, and the model’s end-users are the same GPs. Additionally, this model includes predictors for rarer conditions that will not strongly influence the global measures of predictive performance in Table 1. Other users who wish to apply this model outside the study region may prefer the model derived using the Elixhauser comorbidity categories (B), so these model parameters and their covariances are also supplied in the Supplementary File S1.

Fig. 1 shows the model coefficients and their uncertainties, for the condition category covariates. The values of the mean coefficients are typically larger than two standard errors from zero, which is the traditional definition of ‘statistical significance’. Fig. 2 shows the model coefficients for the patient demographic, modifiable factors and service utilisation covariates.

Fig. 2.  Accelerated failure time survival regression model coefficients related to patient demographic, modifiable factors and health service utilisation, for a model estimating the time-to-next acute admission (2017–21). Some parameters (DeprivationQuintile, HighBMI, LongTermScripts, Cons_12_Count) have been rescaled to facilitate simpler coefficient comparison. See Supplementary Table S1 for full parameter definitions.

Fig. 3 shows interactions between age and service utilisation, and interactions between ethnicity and GP service utilisation. In general, the coefficient on the number of recent practice consultations (global coefficient + whichever interactions are selected) is negative, indicating that an acute admission will, on average, occur sooner as number of consultations increases. This is because frequency of visits to the GP is a proxy for sickness of the patient. However, the interaction between ethnicity and number of recent practice consultations indicates that this effect is slightly less strong for Māori and Pacific patients in the Tū Ora Compass Health PHO, while remaining negative overall.

Fig. 3.  Healthcare service utilisation interaction coefficients for an accelerated failure time survival regression estimating the time-to-next acute admission in the next 6 months (2017–21).

Fig. 4 shows the practice-level variation in model coefficients for the currently active GPs. These coefficients represent variation unexplained by the rest of the model (ie they cannot be attributed to differences in patient demographics or average morbidity), as represented in the demographic and morbidity covariates. They do not necessarily represent changes in quality of care either, however, as the practice with the one of the highest coefficients predominantly services a transient student population who are registered to the practice but may no longer live locally.

Fig. 4.  Practice-level accelerated failure time survival regression model coefficients for the currently active GPs, for a model estimating the probability of unplanned acute admissions in the next 6 months (2017–2021).

To calculate patients’ predicted time to acute admission, one simply needs to take the exponential of the linear predictor. To calculate probability of acute admission in a certain time period, one can calculate quantiles of the time to admission using the mean prediction and scale, then interpolate to calculate the probability of admission in that time period. Results are distributed in this latter form to local GPs in the form of an interactive dashboard that show their patients that have a >50% predicted risk of an acute admission in the next 6 months. As at 1 April 2022, 1212 patients across the Capital & Coast and Wairarapa DHBs have a >50% predicted risk of acute admission in the next 6 months. Of these patients, many will be well-known to their practices, but a minority may not be. Only 92 (7.6%) have been registered with their current practice for <1 year, and 55 (4.5%) have not had a consultation in the last 3 months. Additionally, 631 (52.1%) have at least one named mental health diagnosis, and 28 (2.3%) are aged ≤18 years. These numbers indicate that mental health conditions are a substantial contributor to patient acute admission risk, and that some patients appearing on the list may benefit from proactive preventative care.

The model derived in this study has reasonable performance, with an area under the ROC curve of 0.75 and 0.74 for forecast periods of 6 months and 1 year respectively. These values are slightly lower, however, than other risk stratification models like the PEONY (0.80 for a 1-year forecast)⁵ and QAdmissions models (~0.78 for a 1-year forecast),⁶ although these studies have a smaller patient age range. The model slightly improves on the New Zealand implementation of the PEONY model (0.72 for a 1-year forecast),⁷ and has markedly higher predictive capability compared to a recent Australian model for acute admissions from primary care data (0.66 for a 1-year forecast).¹⁹

The new tool does not specifically address all GP concerns with existing risk stratification tools. It is, however, shown that not all of the highest-risk patients are regular attendees of their practice, so the model outputs may help GPs improve patient care through care plans.

A new empirical risk stratification tool has been developed, to be used in Tū Ora Compass Health practices. The risk calculations are provided to practices alongside each patient’s inputs to the model, to elucidate the reasons for the high-risk assessment. The additional context may help GPs identify patients who are slipping through the cracks, specifically those who may be preferentially utilising hospital services when primary care treatment may be more appropriate.

This model could potentially be generalised to practices outside the regional umbrella of Wellington, Kāpiti Coast and Wairarapa, with some minor adjustment. The practice-to-practice variation is quite strong, so applying the model to new practices would need to derive offset terms if correctly centred predictions are required. However, if only relative risk stratification at a specific practice is of interest, then the model could likely be applied without adjustment. The model would stratify patients predominantly based on the strongest parameters describing age and recent healthcare usage, so would likely still be useful to the first order. Special consideration may, however, need to be given to regional variations in ethnic group and deprivation quintile terms, if these variations are expected to be strong.

The data that support this study cannot be publicly shared due to ethical or privacy reasons.

The authors declare no conflicts of interest.

This research did not receive any specific funding.

Supplementary material is available online.