An evaluation of the quality of evidence underpinning diabetes management models: a review of the literature

Additional keywords: economic evaluation, models of care, primary care, workforce.

Although health systems with strong primary care orientations have been associated with improved access, equity and population health,¹ the primary care workforce is currently facing significant challenges. These include a mal-distribution of resources (supply-side) and health outcomes (demand-side), inconsistent support for teamwork care models and a lack of enhanced clinical inter-professional education and/or training opportunities. These challenges are exacerbated by an ageing health workforce and an ageing general population that has increased prevalence of chronic conditions and multi-morbidity.²

Novel health workforce models have been advanced as a way forward to address these challenges, and growing patient and wider health system demands through new roles (e.g. nurse practitioners), support roles (e.g. allied health assistants) which may substitute for current forms of service delivery, or enhanced roles (e.g. diabetes nurses). Approaches to improve the retention of primary healthcare workers through organisational policies, increase the efficiency of the current skills mix, or improve productivity through linking pay to performance have also been advanced. Although several policy directions have been advocated, there is a lack of evidence about which primary care workforce models are best and what the health effects are for patients.^3,4

The lack of evidence to support these new directions in primary care is surprising, since it is widely acknowledged in medicine (and public health) that evidence-based medicine is essential to reduce the introduction of ineffective and expensive medical treatments.⁵

In this paper, we draw on the National Health and Medical Research Council (NHMRC: 2000, 2001) frameworks for assessing the quality of scientific evidence (and economic evaluations)^6,7 to evaluate the quality of evidence on primary care workforce models for diabetes care.

To make a decision about whether to introduce a new care model it is important to be familiar with the quality of the evidence surrounding that model. In particular, whether the care model (intervention) is likely to achieve the aims of the intended model and whether it represents value for money.⁶

A set of criteria established by the NHMRC (2000, 2001) to assess medical and public health research is used in this paper to appraise diabetes primary care workforce models.^6,7 These criteria are summarised in Table 1. The first is whether the study states clearly what health outcomes may be achieved by the particular intervention, and whether these outcomes have been measured in appropriate units. For example, whether the new primary care model was intended to improve health outcomes or save money but produce no worse health outcomes (non-inferiority). Additionally, the study needs to have a reasonable hypothesis and scientific explanation with evidence (and perhaps a theoretical model) underpinning it, information that can be used to explain why the desired effect is expected from the model.

Table 1.  A framework for assessing the quality of evidence relating to primary care workforce models
Data from NHMRC;^6,7 RCT, randomised controlled trial

The design of studies can vary markedly. The highest quality of evidence, as classified by the NHMRC, comes from systematic reviews of randomised controlled trials (RCTs; Level I). Systematic reviews determine whether a treatment effect can be replicated and, by pooling the results of RCTs, provides a best estimate of the magnitude of the effect.⁷ The second level of evidence comes from a well-designed RCT, which has the strength of minimising bias (Level II). Pseudo RCTs, comparative studies, case control studies or cohort studies have the potential to introduce bias and are ranked as Level III. Case series (pre- and post-test studies) suffer from problems related to the lack of randomisation (e.g. non-comparability of control and treatment groups, different outcome measures for the two groups) and are ranked as Level IV. The lowest level, now excluded by the NHMRC, is expert opinion and consensus from expert committees because these sources do not have a scientific basis (Level V).

Although a study may report a statistically significant effect, it is important to ascertain whether the effect is clinically important. In relation to diabetes care, for example, we would want to know what the cut-off was for scale weight loss to be regarded as clinically important (i.e. losing how many kilograms would be considered as clinically significant). In this case, it would also be important to know whether the difference in this outcome was sustained over time or became insignificant between the intervention and control groups. Although studies may clearly state what the benefits (health outcomes) are for patients, it is important that they also state what the harms (costs) are for patients (and funders). Information on relevant benefits and costs of interventions are needed to undertake an overall (economic) evaluation.

As funding becomes increasingly scarce, due to governments continuously running deficits (as they try to cope with an ageing population) and external challenges (e.g. the Global Financial Crisis, GFC), ensuring that only new healthcare models which represent efficacy as well as efficiency are implemented is crucial.

An appraisal of primary care workforce models applied to diabetes care in Australia published in peer reviewed and ‘grey’ literature between 2005 and 2013 was undertaken. Three main areas were searched:

• key medical databases (MEDLINE, AMED, RURAL, Australian Indigenous HealthInfoNet and The Cochrane Institute)

• key academic peer review journals (Diabetes Research and Clinical Practice, Diabetes Care, Diabetic Medicine, Population Health Management, Rural and Remote Health, Australian Journal of Primary Health, PLoS Medicine, Medical Journal of Australia, BMC Health Services Research, BMC Public Health, BMC Family Practice) relating to diabetes models of care

• several key Australian (Commonwealth and state government) health websites (Australian Government Department of Health, http://www.health.gov.au/; Australian Government Department of Health and Ageing, http://www.yourhealth.gov.au/internet/yourhealth/publishing.nsf/content/home; Government of Western Australia Department of Health, http://www.health.wa.gov.au/home/; accessed 1 July 2014).

Key search terms were ‘primary workforce’, ‘workforce models’, ‘diabetes’, ‘care models’ and ‘Australia’.

The NHMRC’s^6,7 frameworks for assessing evidence (see Table 1) were applied to the studies we found involving primary care workforce models designed to improve the health of adults with diabetes. Of the 14 studies found, only five could be classified as RCTs or systematic reviews and thus correspond to the ‘best’ type of evidence (Level II or I). Nine studies constituted ‘evidence from case series’ or ‘expert opinion’ (Level IV or V). Of the higher-rated studies, one study incorporated an economic evaluation (cost-effectiveness analysis) within the RCT,⁸ whereas another was a systematic review.⁹ Note that the study by Graves et al.⁸ is rated II* rather than simply II because it not only meets the NHMRC’s (2000)⁷ standard for Level II evidence (i.e. a properly designed RCT) but it also includes an economic evaluation.

All studies provided some information about study design and expected outcomes; however, only three studies provided a replicable protocol or detailed information about intervention delivery.^8,10,11 Thirteen studies lacked a theoretical framework. Only four studies provided sufficient detail about the intervention to enable replication.^8,12–14 Only three studies clearly stated the types of changes in health outcomes that equated to ‘clinically important’ ones.^8,10,15

Most studies identified, measured and valued the benefits of their particular intervention; however, only three studies identified, measured and valued the related costs.^8,16,17 Only two of these studies had collected the type of data required to undertake an economic evaluation and thus determine whether the intervention represented value for money compared with the comparator (usual care).^8,17

Table 2 is a summary of the 14 studies found. The quality of each study was rated on the basis of the NHMRC’s (2000, 2001) criteria for assessing evidence (see column 4). The diabetes primary care workforce models are grouped under five headings: one-stop shops, pharmacy models, Aboriginal services, telephone delivered interventions and systematic reviews, which characterise the models implemented. Of the 14 studies, 12 reported statistically significant improvements in several immediate health outcomes (e.g. HbA1c) of patients with diabetes. Only four studies assessed changes in both short- and long-term health outcomes.^8,10,16,17 Most studies used either a small (cut-off was n ~ 100) or very specific populations such as diabetic patients in a rural community.

Table 2.  Summary of primary level models of diabetes care
AHW, Aboriginal health worker; BMI, body mass index; CI, confidence interval; CVD, cardiovascular disease; DBP, diastolic blood pressure; DSS, Diabetes Support Service; ED, emergency department; ESRD, end-stage renal disease; FDS, Freemantle Diabetes Study; FPG, fasting plasma glucose; GGT, Greater Green Triangle; GP, general practitioner; HR-QOL, health-related quality of life; IHD, ischaemic heart disease; IQR, interquartile range; LDL-C, low-density lipoprotein–cholesterol; NA-HARP, Northern Alliance Hospital Admission Risk Program; NPV, net present value; POCT, point-of-care testing; QAAMS, quality assurance for Aboriginal medical services; QALE, quality-adjusted life expectancy; QOL, quality of life; RCT, randomised control trial; SBP, systolic blood pressure; SHDGP, Southern Highlands Division of General Practice; T2D, type 2 diabetes; TC, total cholesterol; TG, triglycerides

Of the seven studies in this group, five were given a quality rating of IV or IV* and two were given a rating of V. The first five studies analysed clinical data from interventions,^{12,13,15,16,18,19} whereas the last two studies (WA Department of Health²⁰ and Foster et al.¹⁴) were primary care policy papers.^14,20 The first five studies provided a sufficient amount of information regarding health outcomes (e.g. H1bA1c and quality of life) to conclude they were both appropriate and relevant. Only one (observational) study also provided information about costs;¹⁶ however, this was limited to the intervention only (as there was no control group) and used to conduct a cost-impact analysis. All studies described the state of existing evidence on the effectiveness of similar interventions in different disease populations¹⁹ or research demonstrating a link between their intervention and the desired effects in diabetic patients. Only the Greater Green Triangle (GGT) diabetes prevention project^12,13 provided details about an earlier implementation of their intervention (pilot). All studies reported changes in clinical outcomes that were statistically significant for patients in the intervention group. Foster et al.¹⁴ only reported ‘outcomes’ on hypothetical case studies, that is, no patients actually experienced the incentives. The transferability of study findings is questionable, because these studies relied on small samples^18,19 or specific populations (i.e. rural communities, socioeconomically disadvantaged groups). Only one study projected the occurrence of health outcomes to several decades.¹⁶

Two studies of pharmacy diabetes care models were found. Both studies were given a high-quality rating: Level II. They shared many of the required criteria for a high rating, such as having a well-defined protocol for the RCT, enabling replication; reporting that the observed improvements in clinical outcomes were greater (and statically significant) in the pharmaceutical care group than in the control; and using a large sample. Clifford and Davis¹⁰ also outlined the type of changes in health outcomes that could be seen as ‘clinically important’ ones for diabetic patients. In addition to assessing changes in immediate outcomes for patients, Clifford and Davis¹⁰ estimated the long-term risk of first cardiovascular disease (CVD) event in patients. Although Clifford and Davis¹⁰ noted the need for cost-effective programs in diabetes care, neither they nor Krass et al.¹¹ conducted an economic evaluation.

Three primary care workforce models for Aboriginal Australians with type 2 diabetes were found. Shephard²¹ and Battersby et al.²² were given a quality rating of IV, whereas McDermott and Segal¹⁷ were given a rating of II. All studies provided a generous amount of information on their particular intervention. Health outcomes were appropriate for addressing the aims of the respective studies. Both immediate (e.g. HbA1c and blood pressure) and long-term clinical outcomes (e.g. avoidable diabetes-related hospitalisations and quality of life) were considered. The studies of Shephard²¹ and Battersby et al.²² are feasibility or acceptability studies and hence report on outcomes relevant to implementation, such as level of satisfaction assessed by patients and health workers. McDermott and Segal¹⁷ conducted an economic evaluation of enhanced coordinated care for patients with diabetes in remote Australian Indigenous communities over a period of 6 years. Shephard²¹ and Battersby et al.²² only assessed changes in clinical outcomes for their patients at 12 months after the intervention; they did not assess the cost effectiveness of their intervention.

Only one telephone delivered intervention was found.⁸ That study is an economic evaluation of a telephone counselling intervention designed to improve the level of physical activity and diet of patients with type 2 diabetes or hypertension in a low socioeconomic area of Queensland. The cost-effectiveness modelling relied on data from a cluster-randomised trial comparing telephone counselling with usual care (brief intervention) and existing practice (real control) collected between 2005 and 2007. Health states were modelled for 10 years after recruitment.

Only one systematic review of international literature on chronic disease management in primary care was found.⁹ Multidisciplinary team and self-management support interventions (and adherence to guidelines) were reported from January 1990 to February 2006 and found to have a significant effect on patients (clinical outcomes). A total of 141 studies and 23 systematic reviews were examined. The study included objective disease control measures, such as patient and practitioner satisfaction and adherence to guidelines. However, the review does not present a summary of cost effectiveness. The main limitation of the study was that it included a large number of models of care (e.g. organisational, professional and financial interventions) using different measures (health outcomes), applied to different patient groups and health conditions. To better assist decision makers, this review should have pooled the results of outcomes for specific patients receiving a well-defined intervention and reported on its relative effect and cost effectiveness over a particular comparator.

The 14 studies of diabetes primary care workforce models appraised in this paper suggest there is an urgent need for more rigorous research on this topic in Australia, especially in relation to the cost effectiveness of new interventions. Only five studies were designed in such a way as to achieve a Level II or I rating, based on the NHMRC’s (2000, 2001) frameworks for assessing scientific evidence. The majority of studies risked the introduction of bias and may have made incorrect conclusions. Several the diabetes primary care models were implemented in quite unique populations (i.e. small, culturally or socioeconomically defined) and hence it is difficult to say whether they (and the possible benefits they describe) could be transferred to the (general) population. Only a few studies described clearly what the intervention and the comparator were and thus could be successfully replicated. Only two studies assessed the cost effectiveness of their intervention compared with (formal) usual care.^8,17

The Commonwealth government is currently piloting new approaches to diabetes management through general practice. It has recognised that benefits can be derived from well-coordinated, integrated, multidisciplinary diabetes care teams and, for this reason, in July 2011 the government committed $30.2 million over 3 to 4 years to a Coordinated Care for Diabetes Pilot in the primary care setting.²³ The pilot is testing a model of prepaid funding for coordinated care of patients with diabetes in general practice.²⁴ With a superior study design it could generate vital information about the relative merits of the intervention in relation to patient outcomes.

It is widely acknowledged that researchers and policymakers need to make decisions based on high-quality evidence;²⁵ however, it is not obvious what effect the evidence is having on primary care workforce reform.

Although there has been an increase in the number of primary care workforce models implemented in Australia, there is a need for more rigorous research to assess: (1) whether these interventions are effective in producing improved health outcomes; and (2) whether they represent better value for money than current practice.

The authors have no conflicts of interest to report.