Accuracy of national key performance indicator reporting from two Aboriginal medical services: potential to underestimate the performance of primary health care

Electronic patient information management systems (PIMSs) have grown in popularity over recent decades as health services transition from paper-based clinical record keeping.¹ As well as recording clinical information, PIMSs are increasingly being used for a range of other purposes, including continuous quality improvement (CQI) and monitoring of health service performance.²

In South Australia (SA), all Aboriginal Community-Controlled Health Services (ACCHSs) have been using the PIMS Communicare (Telstra Health Communicare Systems; Perth, WA, Australia)³ for several years. More so than ever, ACCHSs are looking for new and innovative ways to extract health data from their PIMSs in an accurate and timely manner⁴ to facilitate CQI processes. To enable this, several data-extraction tools (DETs) have been developed. DETs are software programs that run alongside PIMSs to extract clinical datasets. Currently, the most commonly used DET by ACCHSs nationally is the Clinical Audit Tool (CAT; Pen Computer Systems; Leichhardt, NSW, Australia).^5,6

In 2010, the then Department of Health and Ageing initiated the Online Community Health Reporting Environment for Health Services (OCHREStreams) program⁷ in which ACCHSs are required to submit health data against a set of national key performance indicators (nKPIs) on a biannual basis.⁸ Until June 2015, this process involved extraction of nKPI data from PIMSs using the CAT and submission to the Australian Institute of Health and Welfare (AIHW) via the Improvement Foundation.^9,10 For the December 2015 nKPI data submission, the CAT was replaced by the Canning Tool (Arche Health; Perth, WA, Australia) (an alternative DET used for submission of nKPI data to the Improvement Foundation),^9,11,12 but both the Canning Tool and the CAT were used for the June 2016 submission due to several technical issues with the Canning tool.¹³ There was no nKPI submission in December 2016¹⁴ and the nKPI submission process for 2017 is currently under review.¹⁵ Nationally, over 320 primary health care organisations delivering services to Aboriginal and Torres Strait Islander people submit nKPI data via the OCHREStreams program.¹⁶

In 2014, an independent review funded by the Department of Health (DoH) was undertaken to comprehensively assess the data submission pathway of the OCHREStreams program.¹⁷ Importantly, this review did not assess the accuracy of the nKPI data extraction and submission pathway from PIMSs at the health service level to the OCHREStreams Improvement Foundation portal. However, the report did recommend that a data audit exercise be undertaken as a high priority to check the integrity of the extraction process from all compatible PIMSs.¹⁷

Within the Aboriginal community-controlled health sector there is a perception that there are inaccuracies in the nKPI data submitted to the AIHW via the OCHREStreams program. It is important to assess the accuracy of these data given that they are used for individual health service planning and quality improvement activities, and may also be used in the future by the Commonwealth for benchmarking, in addition to informing Aboriginal health policy more broadly. Therefore, the aim of the present study was to assess the accuracy of the extraction and submission pathway for nKPI data for the OCHREStreams program using the CAT from Communicare, the PIMS used by all ACCHSs in SA and up to 60 ACCHSs nationally.¹⁸ The accuracy of the internal Communicare nKPI reporting function was also assessed given the current local use of these reports by ACCHSs for quality improvement as well as for validating nKPI data submitted via the OCHREStreams program.

Two ACCHSs were recruited to the study. Site A is located in a small community classified as very remote and Site B is located within a large town classified as outer regional. Both health services actively participate in the OCHREStreams program.

Given the limited resources available to undertake this project, a sample of OCHREStreams-defined regular clients aged ≥55 years was used; client numbers for this age group were equivalent for both sites. A subset of 13 nKPIs was selected covering risk factor measurement, preventive care and chronic disease management (Appendix I).

A manual case note audit of the nKPI subset within Communicare (version 14.2.168) was undertaken by a clinician at each participating ACCHS. This case note audit acted as the ‘gold standard’ measure against which three automated data extraction (query) methods (detailed below) were compared.

The OCHREStreams program defines that a regular Aboriginal and/or Torres Strait Islander client of the service has had at least three service visits in the previous 2 years.¹⁸ We developed a structured query language (SQL) report (in conjunction with Communicare staff) that was able to compile a list of all patients within the Communicare system for each client who met this definition. This included deceased patients, given the lack of clarity regarding whether or not these patients are included in the OCHREStreams program. The SQL report was run in Communicare by the clinicians undertaking the case note audits and allowed for the inclusion of all current, past, transient and deceased patients, but excluded fictitious patients. The list was then exported into Microsoft (Armonk, NY, USA) Excel and edited to only include patients aged ≥55 years as of 31 March 2015, which remained constant for all reference dates mentioned below.

De-identified patient data were recorded by the respective clinicians within a ‘case note audit’ Microsoft Access database. Fields consisted of date of birth, gender, Communicare ID and their qualification for numerator (denoting the number of clients with an item recorded) and denominator (denoting the number of clients within the population) for any of the 13 nKPIs.

Data from Communicare were collected for three automated nKPI query methods, namely an internal Communicare nKPI report, manual filtering of nKPI data within the CAT and nKPI data from the Aboriginal Health Council of South Australia (AHCSA) Improvement Foundation qiConnect portal, and compared with the case note audit.

Internal Communicare nKPI Reports. Internal nKPI Communicare reports were run for each of the indicators within the nKPI subset by a clinician at each service for OCHREStreams-defined regular clients with the ‘last report date’ set to 31 March 2015. Totals for the combined age groups of ≥55 years were recorded within a ‘query methods’ Microsoft Access database formatted to record aggregated numerator and denominator totals for all three query methods.

CAT Export was used to export a health data subset (dated 31 March 2015) from Communicare into an eXtensible Markup Language (XML) file. This ensured that both the consistency and integrity of the comparative data remained intact because data entered into Communicare after this date by the health service would not affect the XML file.

Manual filtering of nKPI data within the CAT. The XML file was imported into the CAT (version 3.14.2.0) and totals for this query method involved individually constructing the subset nKPIs via the CAT Filter tab. This tab uses cascading filter parameters until the desired indicator definition has been reached. This process was repeated for all subset nKPIs for numerator and denominator totals.

nKPI data from the AHCSA Improvement Foundation qiConnect portal. The clinician submitted the nKPI data to the Improvement Foundation via the Submissions tab of the CAT. The nKPI data that are submitted to the Improvement Foundation by participating AHCSA member health services is automatically compiled into the qiConnect portal, hosted by the Improvement Foundation, as part of a program managed by the AHCSA to support CQI in AHCSA member health services. These data are presented in Microsoft Excel and grouped by nKPI; each nKPI is disaggregated by age group and gender. This export was used to construct nKPI numerator and denominator totals, which were recorded in the ‘query methods’ Access database.

This project was granted ethics approval by the South Australian Aboriginal Health Research Ethics Committee (AHREC) and the Western Australian Aboriginal Health Ethics Committee (WAAHEC).

Results for each of the automated nKPI extraction methods were numerically compared with the case note audit. For the internal Communicare nKPI reports and the CAT nKPI manual filtering, an electronic list of patient IDs was retained so that this could be cross-referenced with patient IDs collected in the case note audit to assess patient status to enable a better understanding of when differences between nKPI totals were observed.

With deceased patients included, the populations aged ≥55 years were identical for the case note audit, the Communicare query and the qiConnect portal for both Site A (n = 51) and Site B (n = 56). If deceased patients were excluded from the case note audit, the population totals for those aged ≥55 years were four less compared with the Communicare query and the qiConnect portal for Site A (n = 47) and two less for Site B (n = 54; Tables c1, 2). The CAT manual query defaulted to exclude deceased and ‘past patients’. With deceased and past patients excluded, totals for the CAT were six less for Site A (n = 45) and 10 less for Site B (n = 46).

Table 1.  National key performance indicator (nKPI) totals for Site A comparing the case note audit (totals excluding deceased patients in parentheses), the Communicare nKPI internal reports, manual filtering in the Clinical Audit Tool (CAT; Pen Computer Systems) and the Improvement Foundation qiConnect portal
MBS, Medicare Benefits Schedule; BMI, body mass index; BP, blood pressure

Table 2.  National key performance indicator (nKPI) totals for Site B comparing the case note audit (totals excluding deceased patients in parentheses), the Communicare nKPI internal reports, manual filtering in the Clinical Audit Tool (CAT; Pen Computer Systems) and the Improvement Foundation qiConnect portal
MBS, Medicare Benefits Schedule; BMI, body mass index; BP, blood pressure

If deceased patients were included, the populations aged ≥55 years with Type 2 diabetes (T2D) were identical for the case note audit, the Communicare query and the IF qiConnect Portal for both Sites A and B (n = 28). However, if deceased patients were excluded from the case note audit, the totals were two less compared with the Communicare query and the qiConnect portal for Site A (n = 26) and two less for Site B (n = 26; Tables 1, 2). Differences recorded for the CAT for Site A (n = 25) and Site B (n = 22) were also accounted for by the exclusion of both deceased and past patients.

As with the client population totals, when deceased clients were included in the case note audit totals and both deceased and past patients were included in the CAT manual query, the totals for all the automated query methods were identical to the case note audit. However, there were two exceptions. First, for Site A, the Communicare nKPI report 6B was different to the other three totals (see Table 1). Analysis of patient IDs showed that the Communicare nKPI 06B report had misclassified one patient in the HbA1c >7% and ≤ 8% group, with the case note audit and other query methods categorising this result as HbA1c ≤7%. For HbA1c results of exactly 7%, conversion to mmol/mol equates to a measure of 53.1 mmol/mol, which, if entered into Communicare, is included in the >7% and ≤8% category. Second, for both Sites A and B, body mass index (BMI) totals were higher for both the CAT manual query and the qiConnect portal (Tables 1, 2). To record BMI in Communicare, patient height (cm) and weight (kg) must be entered into the relevant clinical item and the BMI box must then be selected for the BMI total to be calculated. In contrast, the CAT automatically calculates BMI using the height (cm) and weight (kg) data fields provided and sends these data to the qiConnect portal. Therefore, the totals for the CAT query and the qiConnect portal are more accurate. This also affected the BMI subcategory results for nKPI 12 (Table 1). A further error was found with the Communicare nKPI report for BMI. One result included a BMI measurement that was calculated by Communicare using a weight measurement from >2 years before the audit date. This result was excluded by the CAT.

A further error was detected in the way Communicare calculates blood pressure (BP) results (nKPI 24). For Site B, it was found that one patient who recorded a BP result of exactly 130/80 mmHg was incorrectly classified in the >130/80 mmHg group by the Communicare nKPI 24 report compared with the case note audit. This error was not observed in the CAT or qiConnect portal totals.

Both nKPI 18 and nKPI 19 were omitted from our analyses because totals for patients having received an ‘estimated glomerular filtration rate (eGFR) only’ or ‘albumin : creatinine ratio (ACR) only’ could not be calculated in the CAT. Furthermore, Site B had not yet updated their Communicare nKPI report list, and thus reports for nKPI 18 and nKPI 19 were not available.

By comparing client lists for the population aged ≥55 years with and without T2D and for the individual nKPI totals, we were able to confirm that the same clients were included in the case note audit, Communicare query and the CAT manual query to provide further evidence that any differences observed between the automated query methods and the case note audit were the result of deceased and/or past patients being included or excluded. However, it was not possible to compare patient lists and current client status categories for the qiConnect portal because this only records patient counts. Despite this, we observed that all totals were the same for both the Communicare query and the qiConnect portal (except for BMI and subcalculations for HbA1c). To further explore this, nKPI results for the Communicare query at both sites were compared with the CAT automated nKPI totals (as a proxy for the qiConnect portal totals). This comparison was for all age groups because the CAT automated query was not able to be disaggregated by age group. These totals were also found to be exactly the same and, again, although patient lists and current client status were not able to be compared, the findings provide further evidence that the qiConnect portal includes deceased patients as well.

Measurement of health service performance via OCHREStreams involves the calculation of nKPIs as a proportion of either the total population or a subpopulation, such as people with T2D. Although for some nKPIs included in the present study there were no differences in the totals between the case note audit and the automated query methods (where no deceased and/or past patients were included in any of the totals), the nKPI total (numerator) as a proportion of the total population or the population with T2D (denominator) varied depending on whether deceased and past patients were included in the denominator (see Table 3). This mostly affected nKPIs that are measured over shorter time frames, such as HbA1c (nKPI 5A and nKPI 6A) and BP (nKPI 23).

Table 3.  Site A and B comparisons of national key performance indicator (nKPI) proportions with the inclusion and exclusion of deceased and past patients
Data show the number of patients including/excluding deceased and past patients, with this value expressed as a percentage in parentheses. CAT, Clinical Audit Tool (Pen Computer Systems); MBS, Medicare Benefits Schedule; BMI, body mass index; BP, blood pressure; EtOH, alcohol

For example, the proportion of patients with T2D at Site A who received an HbA1c test within the previous 6 months, as measured by the Communicare reports and the qiConnect portal, was 35.7% (10/28). If deceased patients were excluded from the denominator total, this proportion increased to 38.5% (10/26). If both deceased and past patients were excluded, as with the CAT manual query, 40% (10/25) of patients received an HbA1c test within the previous 6 months (see Table 3). For Site B, the proportions with deceased patients included, with deceased patients only excluded and with deceased and past patients excluded were 28.6% (8/28), 30.8% (8/26) and 36.4% (8/22) respectively.

Most of the other nKPIs where proportions were recorded were also lower for the Communicare report and qiConnect portal nKPI results compared with the case note audit with deceased patients excluded and the CAT manual query. Smoking status recorded (nKPI 09) and BMI recorded (nKPI 12) were exceptions. For smoking status, the time frame for inclusion was ‘ever’ and, as a result, for both Sites A and B, all deceased patients had a smoking status recorded. For BMI, results were higher for both the CAT and qiConnect portal queries due to the automated recording of BMI in the CAT compared with Communicare, as already described. Further, for Site A, the proportion recorded by the Communicare query was slightly better than for the case note audit because all deceased patients had a BMI recorded (because this is a common and simple clinical measurement).

No errors were found in the nKPI data extraction from Communicare by the CAT and then submitted to the qiConnect portal. We found that the Communicare nKPI report included deceased clients and past patients, and we can be very confident that deceased clients and past patients are also included in the qiConnect portal data. This resulted in inflation of client denominators and an underestimation of health service performance as measured by the OCHREStreams system, particularly for nKPIs recording activity in the past 6 months. Minor errors were found in the Communicare nKPI reports related to the grouping of HbA1c and BP results and the calculation of BMI.

When deceased patients were included, the case note audit and the qiConnect portal data results were found to be identical for all nKPIs examined in the present study except those related to BMI. It was found that the CAT automatically calculates BMI if both height and weight variables are present, whereas Communicare requires clinical staff to prompt for a BMI calculation. This resulted in better recording of BMI in the qiConnect portal for nKPI 12 (BMI recorded in the past 2 years) compared with Communicare. Therefore, no errors were found in the accuracy of data extraction by the CAT for the nKPIs examined in the present study at either of the study sites.

Importantly, however, it appears that deceased and past patients are included in the nKPI data submitted via the CAT to the qiConnect portal. It is from here that nKPI data are aggregated and accessed by the AIHW. There is a lack of clarity as to the definition of a regular client for the OCHREStreams program and whether or not it is intended for deceased and past patients to be included. Including deceased and past patients in nKPI data submissions has the potential to underestimate health service performance as measured by the OCHREStreams program. Although this affected most nKPIs in the present study, those measured over time frames shorter than 2 years were particularly affected, such as those relating to HbA1c and BP. It is likely that a similar effect would be observed for other nKPIs not included in the present study.

The effect of the inclusion of deceased and past patients is likely to be far reaching because Communicare is used by more than 60 ACCHSs nationwide,^17,19,20 the majority of which are likely to have deceased people included during any reporting period. Further, although the present study only examined one time point, it is possible that the inclusion of deceased patients has occurred over the life of the OCHREStreams program. If this is the case, over 4 years of submitted data would be affected. The CAT was replaced by the Canning Tool^9,11 for the December 2015 nKPI extraction, although the CAT was used again together with the Canning Tool in the June 2016 submission¹³ (there was no submission of nKPI data in December 2016¹⁴). Given the uncertainty related to future submission of nKPI data via the OCHREStreams program, it is unclear whether deceased people will be included in future nKPI data submissions. Finally, the same deceased patients will continue to be captured in multiple reporting periods if they still satisfy the definition of a regular client of three visits in 2 years.

We found minor errors in the Communicare nKPI reports related to the grouping of HbA1c and BP results and calculation of BMI. Therefore, the use of Communicare nKPI reports has the potential to underestimate health service performance for these nKPIs. This is in addition to the effect of including deceased patients in nKPI calculations, as already discussed. As a result, the querying of the XML file by the CAT for the nKPI subset was more accurate than Communicare’s internal querying methods.

Although the present analysis only focused on OCHREStreams-defined regular clients aged ≥55 years, further investigation into the total populations for the 13 nKPIs aimed to compare the Communicare reports with automated nKPI reports that are available in the CAT, which could not be disaggregated by age. All the CAT nKPI totals were identical to the Communicare nKPI reports, which include deceased patients, and so we are confident that the qiConnect portal data also include these deceased patients. Thus, given the higher mortality rate at younger ages for Aboriginal people compared with non-Aboriginal people, the inclusion of deceased patients will also affect nKPI results for those aged ≤55 years, although to a lesser extent. Further, although we only examined a subset of nKPIs, it is likely that the inclusion of deceased patients will also affect other nKPIs not examined in the present study, although those related to child and maternal health will be less affected. Although we were able to detect several minor errors in the internal Communicare nKPI reports, the small sample size of the present study may not have been sufficiently large enough to detect all internal nKPI reporting errors. In addition, we only assessed internal reporting errors for a subset of nKPIs.

The present study assessed the CAT version 3.14.2.0; as far as we are aware, these issues also affected the CAT–OCHREStreams reporting periods in June 2015 and June 2016.

Further, the present study only investigated the accuracy of the CAT system for Communicare. However, given that Pen Computer Systems use a standardised definition for regular clients, it is likely that deceased and past patients are included for all PIMSs that rely on the CAT for OCHREStreams nKPI submissions.

Finally, the report for the case note audit was developed in conjunction with a senior Communicare software programmer and thus was not completely independent from a system that was being assessed for accuracy by this project. However, the report was independently validated using a fictitious Communicare patient database with the same version of Communicare based at the AHCSA and was found to be 100% accurate.

The present study showed that the CAT accurately extracts a subset of nKPI data from Communicare, which is then accurately submitted to the qiConnect portal (although the CAT is no longer used to extract data for the OCHREStreams program). The inclusion of deceased patients and past patients in the Communicare nKPI reports and the high likelihood that this is replicated in the submitted qiConnect portal nKPI data resulted in health service performance being underestimated for several nKPIs measured in the present study. Given the widespread use of Communicare in health services that submit nKPI data to the OCHREStreams program,^17,19 it is likely that there has been systematic under-reporting of health service performance nationally through this program. Further, minor errors were discovered in the Communicare report coding for some nKPI outcome measures, although the effects of these were overall very small.

From our findings, we make the following recommendations:

• The definition of ‘regular client’ used for nKPI reporting is clarified; ideally this should not include deceased or past patients, which may affect the comparability of future nKPI data collection with earlier reporting periods

• Communicare correct their internal nKPI reporting codes as a matter of urgency

• Other PIMSs need to have similar validation studies performed to assess nKPI accuracy

• The Arche Health Canning Tool needs to be validated for nKPI data submitted to the Improvement Foundation (if it is to be used for future submissions).

None declared.