Analysing aggregate clinical data to support evaluation of the Tackling Indigenous Smoking program, 2014–16

Targeting smokers in health service encounters with repeated unambiguous advice on smoking risks and quit options is effective.^1–3 Screening of smoking status should help identify and target smokers, and helps monitor smoking prevalence in client populations over time. Recent evidence from the Talking About the Smokes study has shown that almost all Aboriginal and Torres Strait Islander smokers (who saw a doctor in the past 12 months) recalled being asked by clinicians whether they smoked, and over 75% recalled being provided with advice or a pamphlet to quit.^4,5 However, other studies involving Aboriginal and Torres Strait Islander people have identified that follow-on referral pathways, including brief interventions and telephone counselling services, are not routinely offered or well targeted for this population.^6–8

In 2016, the Australian Government funded the Tackling Indigenous Smoking (TIS) program, a revision of the Tackling Smoking and Healthy Lifestyle (TSHL) program, which focuses on population health approaches to tobacco control.^9,10 The majority of TIS funding was provided to Aboriginal Community Controlled Health Services (ACCHSs) to develop and implement approaches to tobacco control aimed at preventing the uptake of smoking and to support smoking cessation. In addition to health promotion, many services have enhanced their efforts to deliver smoking cessation support through their clinic services. Although clinical approaches are rarely considered population approaches, clinical interventions can reinforce population measures,¹¹ and health service data on client smoking enhances the ability to monitor changes over time.¹²

This study analysed aggregate service-level data from services funded to deliver primary health care to Aboriginal and Torres Strait Islander people to inform the TIS program evaluation. Our aim was to assess whether TIS-funded organisations have higher recording rates of smoking (process outcome) and decreases in smoking status in clients (impact outcome).

This study was a retrospective cohort study of primary health services for Aboriginal and Torres Strait Islander people, which included 233 services that provide national key performance indicator (nKPI) data.¹³ Data extracted from the client information system is provided via a secure web-based reporting tool, the Online Community Health Reporting Environment (http://www.ochrestreams.org.au/pages/welcome.aspx), and collectively analysed by the Australian Institute of Health and Welfare annually.^14–16 The dataset had not been used previously in an analytical study or in program evaluation.

The broad purpose of nKPI reporting is to provide evidence to: (1) inform policy and program development; (2) monitor Aboriginal and Torres Strait Islander primary healthcare funding; and (3) support continuous quality improvement in the delivery of primary healthcare services to Aboriginal and Torres Strait Islander people. The nKPI data are aggregate data at the service level and include process and outcome variables on three key indicator areas: maternal and child health, preventive health and chronic disease management.

The nKPI data have been collected and reported on since July 2012, with a total of 10 reporting periods (see Fig. 1). There has been an increase in the number of services reporting nKPIs from 90 services in 2012 to 241 services in 2016. Our intention was to analyse data from 2012 to December 2016. However, there were data access restrictions on data from 2012 to June 2014 and the last 6 months of data collected in the 2016 were not available due to a data extraction fault. Hence, analyses were restricted to available data in the following: June 2014, January–December 2015 and June 2016.

Fig. 1.  Timeline of national key performance indicators collected and Australian Government programs (Tackling Smoking and Healthy Lifestyle and Tackling Indigenous Smoking programs) for targeted tobacco control, 2012–17 (grey shading indicates data not available).

The two study outcomes of interest were: (1) the number (and proportion) of clients with a smoking status recorded in the health service records; and (2) the number (and proportion) of clients with smoking status recorded as current, ex- and non-smoker. ‘Exposure’ was defined as an organisation that was funded (n = 44/152) either directly or indirectly (via consortium arrangements) by the Australian Government’s TIS program. Other explanatory variables used in the analysis were remoteness, client numbers, jurisdiction and year of funding.

Services that did not have three reporting periods of data and services in which the definition of regular clients changed over the time periods were excluded from the study.¹⁶ In total, 65% of services (152/233) were included in the study. This included 44 TIS currently funded services and 108 non-TIS-funded services. The nKPI definition of a ‘client’ is a person who has attended the health service three times in the past 2 years.¹⁴ Questions regarding smoking status only apply to adult clients, aged ≥15 years.

Descriptive analysis was undertaken to compare characteristics, such as client numbers, number of services by state or territory and remoteness, by TIS-funded status. Herein, we report total numbers and proportions.

The analytical study assessed the change in the number of clients with a smoking status recorded and outcome of smoking from 2014 to 2016. A negative binomial model (from generalised linear models (GLM), an extension of the general linear model) was used. Negative binomial regression is appropriate where explanatory variables (e.g. interval categorical variables) and errors (random effects) are not normally distributed. Using negative binomial regression allowed us to model aggregate data and to consider interactions between explanatory variables. The GLM function calculated incident rate ratios (interpreted and reported as the reporting ratio (RR)), 95% confidence intervals (CIs) and P-values. All analyses were undertaken in Stata version 13 (StataCorp, College Station, TX, USA) and graphs were prepared in Excel (Microsoft, Bellevue, WA, USA).

Data access permission and ethics approval for the research protocol (Protocol 4/2016) were granted by the Australian Government Department of Health. Results from this study have been discussed with many Aboriginal health services delivering tobacco control campaigns and TIS Australian Government program staff.

In all, 152 services were included in the study, 44 of which were funded for TIS and 108 that were not (Table 1). The location of TIS- and non-TIS-funded services differed by remoteness and location (jurisdiction; Table 1). There was a higher proportion of non-TIS-funded services in remote (and very remote) areas (53% of all non-TIS-funded services; n = 57). Eighty-one services had less than three time interval data points and were not included in the analysis. There were no significant differences between services by client numbers or remoteness.

Table 1.  Characteristics of Aboriginal and Torres Strait Islander primary healthcare services included in the study, Australia 2014–16
Data are given as n (%). TIS, Tackling Indigenous Smoking program; ACT, Australian Capital Territory; NSW, New South Wales

In 2016, 81 187 clients accessed TIS-funded services. This is comparable to client numbers (85 098) at non-TIS-funded services. There is variation in the proportion of clients across state and remoteness. The largest numbers of clients of TIS-funded services were from New South Wales (n = 17 147), the Northern Territory (n = 19 641) and Queensland (n = 28 162).

The proportion of clients asked about their tobacco use among TIS-funded services increased from 85% in 2014 to 88% in 2016. However, the proportion of clients asked about their tobacco use among non-TIS-funded services decreased from 84% in 2014 to 80% in 2016 (Fig. 2).

Fig. 2.  Proportion of adult clients with smoking status recorded by health services according to tobacco control funding of Aboriginal Community Controlled Health Services in Australia 2014–16. TIS, Tackling Indigenous Smoking program.

Among TIS-funded services, the tobacco use RR was 1.58-fold higher (95% CI 1.30–1.91; P < 0.001) after controlling for location (jurisdiction) and remoteness (Table 2). The highest change in assessing tobacco use was among TIS-funded remote services (RR = 6.55; 95% CI 5.18–8.27; P < 0.001).

Table 2.  Recording of smoking status by Aboriginal and Torres Strait Islander primary healthcare services according to Tackling Indigenous Smoking (TIS) funding, 2014–16
Data are given as the reporting ratio (RR) with 95% confidence intervals (CIs) in parentheses

In 2016, the first year of the TIS program, TIS-funded services were also more likely to record smoking status among clients than non-TIS-funded services (RR = 1.11; 95% CI 1.00–1.28; P < 0.001).

Over half of all clients (53% in 2014, 2015, 2016) attending all services were reported as current smokers (Fig. 3). Smaller proportions were reported as ex-smokers (14% in 2014 and 2015, 15% in 2016) and just over one-third were reported as non-smokers (33% in 2014, 32% in 2015 and 2016). TIS funding was not associated with any change in reporting of clients as current smokers, ex-smokers or non-smokers across the three reporting periods (Table 3).

Fig. 3.  Proportion of adult clients with smoking status recorded by Aboriginal and Torres Strait Islander primary health care services according to Tackling Indigenous Smoking (TIS) funding, 2014–16.

Table 3.  Smoking status outcomes reported by Aboriginal and Torres Strait Islander primary healthcare services according to Tackling Indigenous Smoking (TIS) funding, Australia 2014–16
Data are given as the reporting ratio (RR) with 95% confidence intervals (CIs) in parentheses

The results of this study show that services funded under the TIS program (and prior TSHL program) are asking about their clients’ tobacco use at a statistically significant higher rate than non-TIS-funded services after controlling for differences due to client numbers by service, jurisdiction and remoteness (Table 2).

Recording of tobacco use has been identified as an important first step in a health systems approach to facilitate tobacco dependence treatment.¹⁷ Several studies have identified improvements to the recording of tobacco use in clinical populations after policy emphasis at the clinic level.^3,18 Although not specific to the Indigenous context, health systems approaches are considered to provide tobacco cessation support and improve tobacco cessation outcomes.¹⁹ Assessment of tobacco use is an important initial step in enhancing the clinical focus to reduce the harms of tobacco use in clients and is likely to lead to better monitoring of tobacco use over time.

We cannot attribute this finding directly to TIS funding because of the study’s selection bias and the broader tobacco control policies, however services integrating procedures for tobacco control incorporating health promotion and clinical approaches together may realise improved cessation outcomes over time. Although outside the scope of the present study, follow-up identification of services that integrate clinical and health promotion work is an important future research area.

Overall, there were high levels of recording smoking status in clients across most services in this study (>80% of clients had a smoking status recorded). Although the frequency of updating smoking status cannot be determined due to variations in practices at the service level, if identified smokers are followed-up with brief intervention or quit advice and referrals to intensive support at every visit, it is likely that better outcomes in smoking cessation will eventuate among those clients.^20,21

However, we found no change in smoking status outcomes in either group (TIS funded vs non-TIS funded). This is potentially not surprising given that there were only 3 years of data available for analysis. In Aboriginal and Torres Strait Islander population-level studies with data from over 10–20 years, the declines in smoking prevalence are important but modest, in the order of 1–1.5% per year.^22,23 At this early stage of the revised TIS program in 2016, many services funded were still in the recruitment phase, writing their action plans, training staff and gearing up for implementation of activities and campaigns.²⁴ It is likely that changes in smoking prevalence need longer periods of time and will not be realised until much later in the delivery phase; such lag time issues are well known in health program delivery.²⁵

Given that this study included only 3 years and the first 6 months under the revised TIS program (2016 data), the indication of higher recording of tobacco use is important for future program monitoring. Further, our results show that remote services funded through the TIS program had increased reporting rates of clients’ smoking status than all other services. This could be the result of increased awareness among health professionals and service priorities in remote areas towards addressing tobacco use in these areas, particularly because smoking prevalence has not changed.²⁶ Given that remote areas are not well serviced in terms of outside referral pathways for quit support,⁶ this process indicator in recording of smoking status is important.

Examining nKPI data did not require significant investment or additional resources and extended the purpose and value of routinely collected data. Using these data as part of an evaluation process can reduce the burden of data collection and analysis on services that are primarily responsible for service delivery and program implementation. However, using these data only presents a partial picture on the effectiveness of work undertaken in tobacco control, and there were significant limitations in the methods.

In particular, the study had an inherent selection bias because the 2016 TIS funding was a targeted funding round to selected services based mostly on those previously funded by the TSHL program and services considered to have greater capacity to deliver tobacco control campaigns and activities. This substantially biased the results due to lack of randomisation. As such, the services selected for funding may differ from those not funded by the TIS program. These TIS-funded services may have had greater motivation and organisational support to intervene at a population and clinical level and may, overall, be better at reporting across all nKPI data items. Other confounders may include staff numbers, overall budget numbers, proximity and number of other services in the region and other service dynamics, including the support given to motivate clinicians to act. A large number of services (n = 81) was excluded from the analysis because of missing data (n = 50) or due to changing definition of a ‘client’ (n = 31). As is often the case with real-world studies, such limitations are unavoidable but suggest that caution should be applied when interpreting the findings.

There are also some study limitations due to a lack of consistent definitions around the smoking status data collection. We are unsure how often smoking status is updated and this is not included as an nKPI indicator, which means that smoking status may have been recorded a long time ago and not updated. The category of ‘non-smoker’ in nKPI data is also problematic because it could potentially include ex-smokers and never smokers; this should be revised to ‘never smokers’. Some client health record systems cannot record when smoking status was last asked (as such, it is currently recommended that this be assessed at every visit). The study was further restricted due to lack of access and availability of all years of nKPI data. Data restrictions on earlier nKPI data 2012–14 and the time frame of this study meant that longitudinal analysis could not be undertaken. There would be value in considering analysing data in the earlier years, because reports on nKPIs by the Australian Institute of Health and Welfare found recording of smoking status in clients has increased over time from 64.1% in 2012 to 80.1% in 2015.¹⁶

Individual client record data would improve the statistical power of this study. However, one of the purposes of the approach to this study was to use existing administrative data for monitoring and evaluative purposes. The Department of Health owns the aggregated nKPI data,¹³ and the ownership of individual data on clients remains with each health service. It may be worthwhile for a group of services contributing to the nKPI data to collectively consider the value of pooling their individual client data, such as has occurred in earlier studies by the Queensland Aboriginal and Islander Health Council.²⁷ As with previous research,²⁷ such data collection and analysis is likely to need an appropriate Indigenous organisation to govern the data and the analytical approach.²⁸

Although the TIS program emphasises a population health approach, most TIS-funded organisations are health services that have integrated enhanced clinical efforts or counselling as part of their tobacco control measures, therefore justifying analysis of client data.²⁹ Further, the follow-up of identified smokers with clinical interventions, such as brief interventions and advice on pharmaceuticals that support quit attempts, has been associated with reduced client smoking rates.³⁰ Clinical interventions are often seen outside the population health approach, but we suggest an alternative approach is that they be considered within a health systems approach, providing supportive action to enable quit attempts and reinforcing health promotion messages and educational campaigns.³¹

As such, we recommend that analysis of the nKPI data should be one part of the TIS program monitoring in future. Our study’s early indications of change between those services funded under the TIS also require further service-based data and discussion among services. The ongoing analysis of nKPI data should only be considered alongside other data sources and field-based evaluation, particularly because of the very limited number of client smoking variables in the nKPI dataset. Revisions to the nKPI data could consider additional variables that record process and behavioural outcomes on the casual pathway to smoking cessation among the health service clients, including quit attempts and referrals to Quit support services.

Even with these additions, any analysis of nKPI should be supplemented with other data sources in evaluating programs like the TIS. Other data are particularly important given the population health focus of the TIS program. A closer examination of associations between changes in normative and social behaviours of smoking among the Aboriginal and Torres Strait Islander population and the TIS program is required.^8,32,33 Although clinical services and associated data are important and can provide some indication of change, the TIS program is not solely about supporting clients in the clinical setting and nKPI data have reduced value for measuring population, community and regional health outcomes.

This study found higher reporting rates of client smoking status in services funded under the Australian Government’s TIS program. This may be linked to increased tobacco control resources and priorities to address tobacco in those services funded through the TIS, but the study’s selection bias highlights the difficulties in evaluating programs that have targeted funding rounds. Importantly, extending funding out to services that are not achieving results in these nKPIs may be just as important as drawing comparisons between TIS-funded and non-TIS-funded organisations.

The public health importance of this study has been in establishing the analytical process for aggregate health service data in nKPIs and assessing the validity and limitations of using this data source for population health program evaluation. This study’s value will be enhanced if continued analysis is applied to future data waves.

The authors declare no competing interests.