Using run charts for cardiovascular disease risk assessments in general practice

KEYWORDS: Cardiovascular diseases; risk assessment; primary care; run charts; quality improvement

In New Zealand, cardiovascular disease (‘CVD’) guidelines recommend screening for CVD and diabetes in all men aged >45 years, all women aged >55 years and 10 years earlier for people who are of Māori, Pacific or South Asian ethnic groups, or have known CVD risk factors.¹ Management recommendations are based on five-year CVD risk stratification, with greater the intensity of lifestyle and medication management recommended for people with higher estimated CVD risk.¹ If fully implemented, this targeted approach could reduce future CVD events by an estimated 50% or more.²

To encourage CVD risk assessment, the Ministry of Health introduced a primary care performance target in 2012: to achieve 60% of the eligible population having a CVD risk assessment (including screening for diabetes) by July 2012, 75% by July 2013, and 90% by July 2014.³ The target was accompanied by a modest financial incentive paid to primary health organisations (PHOs) and public benchmarking of PHO performance. In 2012, we estimated that approximately half of the eligible population from the Northland and Auckland regions had been risk assessed (Wells, S. unpublished). To support practices to achieve their targets we considered that run charts may be useful. A run chart is a simple visual display of data over time that provides a dynamic view of the performance of a care process within a health service.^4–6 Run charts are also used to determine if interventions have resulted in improvement and if improvements are sustained.⁷ Run charts can be applied when other methods to determine statistical significance are not useful.⁵ Run charts have been used in hospital and ambulatory settings for some time to monitor quality improvement activities^6,7, but their use in New Zealand general practice is not widespread.

The aims of this study were to investigate the feasibility and acceptability of run charts that display weekly CVD risk assessments in a group of Northland general practices and to determine whether this tool might help to increase the rate of eligible, enrolled patients having a CVD risk assessment.

The study was conducted in Northland general practices from November 2013 until 31 July 2014. Primary health care in Northland is delivered by 39 general practices belonging to two PHOs – Manaia and Te Tai Tokerau. The PHOs share common information services, population health monitoring and clinical directors. In addition, the PHOs employ facilitators who support individual practices in achieving population health targets. All except one Northland primary care practice use a web-based computerised decision support tool (PREDICT)⁸ to conduct CVD risk assessments for their eligible population. CVD risk profiles of individual patients are sent via secure messaging to the PREDICT server and within seconds a CVD assessment score is calculated and returned to the practice to be saved in each patient’s electronic medical record. Encrypted risk profiles for the two PHOs are also stored on a secure server housed by Enigma Solutions Ltd on behalf of the two PHOs. Practices can check their weekly counts of CVD risk assessments (and estimated proportion of the eligible population assessed) if they choose. The investigators received permission from the study PHOs to access anonymised aggregated count and proportion data stratified by practice.

At the time of the study 20 practices in the PHOs already participated in the EPOCH trial, a randomised controlled trial of point of care testing (POCT) for HbA1c and cholesterol. As the primary outcome of the trial was completed CVD risk assessments in eligible patients, the trial 20 practices were excluded as run charts would represent a co-intervention making it difficult to tease out the impact of each intervention separately. After excluding one practice not using PREDICT, investigators asked the remaining 18 practices if they would be interested in trying out run charts. Nine practices accepted this invitation; the remaining nine practices provided the control group.

The main study outcome was the weekly proportion of eligible patients with completed CVD risk assessments. Weekly counts of completed CVD risk assessments were compared for the nine intervention and nine control practices. To determine whether changes in risk assessment may be attributable to the run chart, weekly data points for every practice were collected for 19 weeks before introducing the run chart (i.e. baseline practice performance). These data (from 14 Nov 2013 to 20 Mar 2014) were plotted on a paper run chart for each practice (Figure 1). For the intervention practices, PHO facilitators visited and provided brief instructions on how to fill them in and to continue plotting data for the next 19 weeks (from 27 Mar 2014 to 31 July 2014). The control practices did not receive a run chart or the instruction visit. At the end of the study, investigators (CT and SW) visited intervention practices and interviewed practice staff.

Figure 1. Example of a Northland Practice Run Chart

We used a random coefficients model to determine whether there was a change in rate of CVD risk assessment (slope) from pre- to post- intervention by aggregating the mean slopes from the nine practices in the intervention group and comparing that to the mean slopes achieved by the nine control group practices. The unit of analysis was the practice and the percentage of eligible patients with a completed CVD risk assessment was the primary outcome. The model has both fixed effects and random effects. The fixed effects were the changes in percentage screened pre- and post- intervention and interaction terms were used to examine differences of the slopes of the intervention and control group across time. Random effects included the intercept and slope for each practice with a first-order autoregressive covariance structure. The analyses were conducted using SAS software version 9.4 (SAS Institute Inc., Cary, NC, USA.).

We identified a purposeful sample⁹ of practice staff taking run chart ownership or who were practice champions of the CVD risk assessment target. We aimed to conduct interviews with at least one staff member in each intervention practice, either face-to-face or via telephone. One of the authors (CT) invited clinical staff to be interviewed and after receiving consent an interview time was scheduled. The semi-structured interview schedule was limited to three questions: (1) How did you use the run chart? (2) What could be improved? (3) What changes did you make (if any) as a result of having the run chart in the practice? At the time of the interview, field notes were taken including as many verbatim comments as possible and checking with interviewees that these were correct. The field notes were transcribed immediately afterward each interview. Data from the interviews were then analysed by a general inductive approach, similar to grounded theory.¹⁰ The authors (SW, CT) identified emergent themes and relationships after iterative reading and discussion. All potentially identifying information was masked to protect participant confidentiality.

This study was approved by The University of Auckland Human Participants Ethics Committee, reference 2014/011204.

Weekly proportions of each practice’s eligible population having CVD risk assessments are shown in Figures 2 and 3 for intervention group and control group practices, respectively.

Figure 2. Intervention Group: Weekly proportions of the eligible population with completed CVD risk assessment from November 2013–July 2014

Figure 3. Control Group: Weekly proportions of the eligible population with completed CVD risk assessment from November 2013–July 2014

Initially, the nine practices agreeing to trial the run charts had a wider spread than control practices in the proportions of eligible patients CVD risk assessed. As the weeks progressed, the proportion of the eligible population with completed CVD risk assessments increased so that by the end of the study period all practices (intervention and control) clustered around 90% completion. Four of the nine run chart practices and six of the nine control practices improved CVD risk assessment proportions in the four months (March to July 2014) post intervention. One practice in each group showed continuous improvement throughout the study period.

While there was a step change at week 19, the random coefficient model (Figure 4) showed no significant difference in the slopes between the two groups; the mean difference of pre-run chart slope and post-run chart slope in the intervention group was 0.03% more CVD risk assessments per week, while the mean difference in the control group was 0.07%. The estimate of between group difference was 0.04% per week (95% CI: –0.26 to 0.35, P-value = 0.77).

Figure 4. Mean slope of completed CVD risk assessment in intervention and control practices before and after the intervention practices received a run chart

Five practices in the intervention group and four practices in the control group had less than 80% CVD assessment at week 1. A subgroup analysis of these practices also showed no difference between the two study groups (95% CI: –0.41 to 0.41, P = 0.9996).

Investigators SW and CT interviewed four nurses, five practice managers and two GPs. Of the nine practices with run charts, seven had plotted points on the chart with six plotting weekly data points consistently over the entire period. After data analysis, three themes were identified; a run chart is an easy to use quality improvement tool; it serves as a visual reminder for measuring progress; and every tool needs easily available data and a champion.

All practices found run charts simple to understand, uncomplicated and straightforward to use. The only improvement identified was possible enhancement by using coloured pens. The six practices using the charts extensively (weekly plotting) had placed them in a highly visible place (e.g. staff tea-room or by the phone in the practice managers’ room or nursing station). Displaying charts in a place where all could see them ‘got a bit of buy-in’ and created a ‘little bit of buzz’. One Nurse Manager noted that she would be reminded by other members of the team to fill it in.

‘a simple way for everyone to see where you are at’ (Clinical Nurse Manager)

Charts were described as ‘useful as a visual prompt’ and provided a graphical communication of the journey to date. In one practice, the run chart was stuck onto the wall of the practice manager’s office. All staff ‘knew it was there and what it was for’. From her perspective, it made the target to 90% easier as people could see how many CVD risk assessments they needed to complete each week- ‘what to pitch at’- so it made the task more achievable. She reported that because of the run chart, staff were ‘more onto it - more regularly’. Four managers reported that they used the run chart as a prompt for discussion in team meetings.

‘(we would discuss) how we are going, what we need to do, who do we need to monitor…talk to the team about reasons ….while someone in the whanau may need a CVD risk assessment they were also thinking what else do we need to do, who in the whanau is due, for example, a smear.’ (Clinical Nurse Manager)

The charts supported ‘not just a one-person effort’ but a whole of team approach towards achievement of the CVD risk assessment target goal.

‘…the whole team got behind it-including the doctors’ (Practice Manager)

In one practice there was speculation that it might be useful for patients to see the charts as well, to show that the practice was ‘kind of working on improving care’.

Interviewees observed that run charting CVD risk assessment performance would not be easy without health information technology tools helping to identify individual eligible patients and aggregating completed CVD risk assessments. Furthermore, the proportion of completed assessments could change rapidly with shifts in the eligible denominator population with new patient enrolments and patients becoming newly eligible (previous CVD risk assessment occurred 5 years plus one day ago).

These tools allowed practices to understand how many patients they needed to screen each week.

‘… we knew how many a day - 14 virgins this week!’ (a virgin was a never-screened eligible patient). ‘We pulled patients out of the woodwork to get CVD RA [risk assessment] done!’ (Clinical Nurse Manager)

Run charts were useful in practices where the designated CVD risk assessment champions found a place in their processes for them; as a visual reminder, a prompt for on-going conversations and graphical depiction of their commitment to the target goal. Three practices did not use them. Two of these three practices used other tools to generate weekly task reminders (their weekly list of people to contact, to organise blood tests, etc.). For these two practices, filling in a run chart was ‘just another job’ and a ‘waste of time’. In the third practice, the practice manager reported that the run charts ‘were good to start out with but dropped off’. She said that the time for administrative tasks in the practice was at a premium and that there were ‘too many things to do in a small practice’.

Run charts were useful for six of nine intervention general practices in this observational study and viewed positively as a means of performance monitoring. Key findings were that they were easy to use, provided a useful visual prompt and helped to facilitate a ‘whole-of-practice’ team approach to CVD risk assessment. Their usefulness relied on having easily available and trusted data. Collection and graphical display of data every week made the target ‘more do-able’ and enabled teams to see that changes in practice were related to improvements in care processes.

We found no difference in CVD risk assessment performance between intervention and control groups. Benefits from interventions are more likely to be reported where there is a sizeable evidence-practice gap at baseline. In this study all but two practices had already risk assessed most of their population. A common limitation of observational studies is the influence of background changes on the outcome studied. Associated with other external pressures such as open benchmarking, the Ministry of Health target of 90% risk assessed by July 2014 was a powerful modifier of practice behaviours. The timing of this research coincided with other initiatives (the two PHOs actively monitored and supported their member practices to achieve the target) that had already been implemented in all practices, possibly swamping any additional effect run charts may have had.

There are no published studies on the use of run charts in primary care for CVD risk assessment. However, due to their simplicity and simple rules for detecting non-random patterns and variation over time, the run chart has been described as the ‘universal tool’ for virtually every improvement project.¹¹ They have been employed by clinical teams for a wide variety of health care processes such as HIV screening in primary care,¹² for inpatient bronchiolitis,¹³ chemotherapy,¹⁴ measuring inpatient harms,¹⁵ central line associated bacteraemia,¹⁶ orthopaedics,¹⁷ acute coronary syndrome,¹⁸ inhaled corticosteroid prescribing,¹⁹ insulin therapy²⁰ and also by patients.^5,21 At the time of the study, the two PHOs also used run charts to track immunisation and to illustrate progress towards population health priority areas such as giving brief advice for smoking cessation. From our interviews with intervention practices, run charts were used as a visual tool only, rather than in conjunction with other quality improvement methods such as rapid cycle improvement (e.g. Plan-Do-Study-Act cycles) or fish bone diagrams, Pareto charts and value stream mapping.

This research is limited by its design: a non-randomised intervention study with a small number of practices. Intervention and control practices may have differed in ways that affected their CVD risk assessment process. However, we used a contemporaneous control group. Without a control group, there could have been erroneous assignment of an intervention benefit on the outcome. Three of the nine control practices had long achieved the 90% target and the six others improved dramatically, indicating prolonged and sustained efforts related to other practice and PHO interventions. A further strength is the measurement of performance over 19 weeks before giving the practices the run charts. It is recommended to collect at least 12–16 data points in a run chart to understand the ‘voice of the process’ before introducing a change.⁶ In our study most of the intervention practices were already improving performance before receiving a run chart and, like the control group, there was a considerable ceiling effect.

Our collection of qualitative feedback on the use of this quality improvement tool in practice and its impact on practice teamwork sheds further light. Run charts were feasible in everyday general practice and helped team work. Primary care clinicians can learn a great deal about their performance by using a simple run chart that only requires paper and a pencil, the relevant data and minimal mathematical complexity. To determine whether a process is stable, improving or getting worse requires only application of some simple rules after calculating a median line.^5,11

While the use of run charts did not make a measurable difference in CVD performance, as one manager said; ‘it made a difference- it mattered to us’.

Prof Chris Bullen, Dr Natasha Rafter and Ying Huang have no competing interests to declare. Dr Kyle Eggleton is a Director of Manaia PHO. Catherine Turner was the Population Health Strategist and Analyst for Northland PHOs. Dr Susan Wells is the HRC-funded VIEW primary care cohort leader. This cohort is generated via the use of the PREDICT programme in primary care.