Evaluation of the system of surveillance for enteric disease outbreaks, New South Wales, Australia, 2000 to 2005

Surveillance systems are designed to identify, investigate, control and report rapidly on outbreaks of disease, and to identify factors that can help prevent future outbreaks.¹ Enteric disease outbreaks have been notifiable in NSW since 1991, and surveillance has been progressively improved from 2000 onwards with the introduction of OzFoodNet, a collaborative national network for the surveillance of enteric disease.² This network aims to provide better understanding of the causes and incidence of foodborne disease in the community, and an evidence base for policy formulation.²

A description of the notification and reporting requirements for enteric disease outbreaks in NSW is provided in Table 1. In NSW, public health units (PHUs) are responsible for investigating notifications of enteric disease, and providing summary reports of enteric disease outbreaks to the NSW Department of Health using standard reporting forms. Outbreaks may also be detected through review of routinely collected notifiable diseases surveillance data, the NSW Food Authority complaints hotline, or reports from clinicians, institutions or laboratories, and members of the public (Figure 1).

Table 1.  Notification and reporting requirements for enteric disease outbreaks in New South Wales

The NSW enteric disease outbreak surveillance system collects data from all NSW public health units; it therefore covers the entire population of NSW. The specific objectives of the NSW enteric disease outbreak surveillance system as described in the NSW Health Notifiable Diseases Manual are to:

• identify the source of the outbreak

• prevent further cases

• monitor the epidemiology to inform the development of better prevention strategies

• monitor the development of unusual or emerging pathogens

• fulfil international reporting requirements.³

We undertook the first evaluation of the NSW enteric disease outbreak surveillance system, which aimed to:

• determine whether the objectives of the system are being met

• evaluate the specific attributes of the system

• identify areas for improvement.

The evaluation was based on the Centers for Disease Control and Prevention guidelines for evaluating surveillance systems (Table 2).^4,5 The public health importance was derived from the results of the accompanying enteric disease outbreak review presented in this issue.⁶

Table 2.  Tasks for the evaluation of a surveillance system

The simplicity, flexibility and acceptability of the system were examined through unstructured interviews with key Communicable Diseases Branch (CDB) staff, and by detailing the flow of information through the system (Figure 1). Other system attributes were examined by analysing the summary reports from the two NSW enteric disease outbreak databases (the Gastroenteritis in Institutions Database and the OzFoodNet Outbreak Summary Database), where symptom onset for the first case occurred between 1 July 2000 and 30 June 2005.

To assess the completeness of enteric disease outbreak reporting by PHUs, we calculated the proportion of all outbreaks first reported in any form (including phone calls, emails and other initial reports) to the CDB from January to December 2005 that had a completed final outbreak summary report. Timeliness of reporting was evaluated by determining the period between the date of symptom onset for the first case and the date of the final summary outbreak report.

Completeness was assessed by examining the completeness of key data fields, including: onset date for the first case; summary report date; number of people at risk; number of people affected; number of hospitalisations; number of deaths; and number of clinical specimens collected.

The usefulness of the system was examined by reviewing the objectives of the surveillance system, interviewing key informants from the CDB enteric diseases team and reviewing the output of the system, including reports and policy interventions. Policy outputs for the period 1 July 2000 to 30 June 2005 were identified in past editions of the NSW Public Health Bulletin and through interviews with the CDB staff.

Enteric disease outbreaks in NSW were identified and reported through several mechanisms (Figure 1). The maintenance and integration of the flow of outbreak information along these pathways was primarily a manual process involving considerable work by staff from notifying laboratories, PHUs, CDB and the NSW Food Authority. There were few automated components to this process. The CDB enteric diseases team entered all data from outbreak summary forms received from PHUs into the relevant enteric disease outbreak database – either the OzFoodNet Outbreak Summary Database or the Gastroenteritis in Institutions Database – as there was no mechanism for PHUs to enter their own data.

Key informants reported that these databases only captured final summary outbreak information and provided no mechanism to track the course of outbreaks from initial identification through to completion. Some outbreaks were incompletely reported and others were not entered into the database. The system could not readily capture and organise the large amount of information generated throughout the course of an outbreak investigation in a timely fashion. The outbreaks reported ranged in size and scope. The surveillance system catered for this; however, the databases were not flexible enough to incorporate additional information where required, and data from other Australian states could not be readily accessed.

It was not possible to determine the potential relationships between individual disease notifications in the NSW Notifiable Diseases Database and enteric disease outbreaks recorded in the enteric disease outbreak databases. In addition, cross-checking whether cases identified in outbreaks had also been entered into the NSW Notifiable Diseases Database was not possible, as information on individuals was not captured as part of the outbreak summary reporting process.

The CDB staff who managed and maintained the surveillance system for enteric disease outbreaks reported that it was cumbersome and labour intensive. The system for reporting enteric disease outbreaks created a large paper trail and involved many hours of data entry work, particularly during epidemic winter seasons of viral gastroenteritis. The outbreak summary forms provided by PHU staff often required interpretation by the CDB enteric diseases team, and required additional work to gather missing information where forms were incomplete.

Completed outbreak summary forms were mailed, sent by secure fax or emailed to the CDB. Individual case identifiers were not included in the data forms; information was reported in summary form only. Information on individual institutions and businesses with suspected or confirmed outbreaks was included, highlighting the need to ensure security of data at all times. With no system for tracking outbreaks, and the large paper, email and fax trails generated by the outbreak reporting system along the numerous reporting pathways, there was considerable potential for information to be misplaced or lost.

The true representativeness of the surveillance system could not be evaluated because this can only be measured through comparison to the true rate of enteric disease outbreaks in NSW, which is not known. An approximate assessment of representativeness can be made by comparing the rates of enteric disease outbreaks across different areas of NSW, assuming that the occurrence of enteric disease outbreaks do not vary by location. The population-adjusted public health unit total rate of reporting of enteric disease outbreaks was highly variable, ranging from 0.5 to 7.4 per 100 000 population.⁶

The completeness of the surveillance system was judged by assessing the proportion of critical data fields that were completely collected for all outbreaks during the period of the evaluation. The outbreak data were almost 100% complete for the date of onset of the first case and the number of people affected. Completeness for other important fields, such as the number at risk, the number hospitalised, the number of deaths, the number of stool specimens and the summary report date, was variable both over time and between public health units (Table 3). In addition, CDB staff reported that it was common for data on the environmental component of foodborne disease investigations to be incomplete, and that they were unable to monitor the completeness of outbreak reporting or be sure that they had received all essential outbreak data.

Table 3.  Completeness of enteric disease outbreak data field by reporting unit, New South Wales, 2000–2005

During 2005, final summary outbreak forms were received for 132 (63 per cent) of 209 provisionally reported outbreaks, including summary reports for 49/116 (42 per cent) of foodborne outbreaks and 83/93 (89 per cent) of institutional outbreaks. Some of the outbreaks provisionally reported may not have been considered to warrant further investigation or summary reporting by the PHU.

Sensitivity could not be evaluated, as data on the true total number of enteric disease outbreaks in NSW is not available. Positive predictive value could also not be formally evaluated, but all of the outbreaks reported using the final summary reporting form appeared to be true enteric disease outbreaks, as no non-enteric pathogens were identified.

There was inevitably a delay between the date of onset of symptoms in the first case and the final date of the outbreak summary report. The median time to summary reporting of all enteric outbreaks over the 5-year period was 20 days (inter-quartile range 6–52 days). For foodborne outbreaks, the median time to a summary report was 32 days (inter-quartile range 10–105 days), and for non-foodborne outbreaks 19 days (inter-quartile range 6–47 days). The time to summary reporting of foodborne outbreaks was significantly longer than for non-foodborne outbreaks (p < 0.001, Kruskal–Wallis test), although the median time to summary reporting for all outbreaks was well within the required 30 days since the last outbreak case was identified.

As the period of evaluation covered the introduction of two outbreak databases, one during early 2000 and another in 2003, the methods for surveillance and reporting of enteric disease outbreaks changed over the five-year time period investigated, both in terms of the reporting forms used, data fields collected and databases used. This meant that some data were incomplete or not completely comparable, even over this relatively short time period.

The cost of the surveillance system was not assessed due to time constraints.

Between 2000 and 2005, 998 enteric disease outbreaks were reported (148 foodborne and 850 non-foodborne), affecting 24 260 people, and associated with 771 hospitalisations and 21 deaths. The outbreaks reported during the evaluation period are described in more detail in ‘Enteric disease outbreak reporting, New South Wales, Australia, 2000 to 2005’ in this issue.⁶

Despite the limitations of the surveillance system, the key informants indicated that they were able to use the enteric disease outbreak surveillance system data to produce useful information, such as: the incidence of outbreaks in NSW; assessment of the success of outbreak control efforts; identification of the probable cause of an outbreak; and identification of measures that could contribute to more effective prevention of enteric disease outbreaks.⁶

The CDB users of the system, and the results of an analysis of the information available from the enteric disease outbreak databases, indicated that the surveillance system did provide a mechanism for meeting all the objectives of the surveillance system at some level.

Policy outputs of the NSW enteric disease outbreak surveillance system were difficult to identify due to an absence of a system for identifying, organising and recording policy outcomes of outbreak investigations. Products of the surveillance system that were identified through document review and interviews with key informants included:

• fortnightly, quarterly and annual NSW Health and OzFoodNet reports (published as part of the national OzFoodNet reports in Communicable Diseases Intelligence)

• summary institutional outbreak data published in each issue of the NSW Public Health Bulletin and available from the NSW Health website (www.health.nsw.gov.au)

• ad hoc provision of outbreak data as requested by other jurisdictions, government agencies, research institutions and industry

• information that contributed to policy development within the NSW Department of Health and the NSW Food Authority, and also to national foodborne illness and health policies

• information that contributed to advocacy measures, including media releases (Table 4).

Table 4.  Examples of policy outputs from enteric disease outbreak investigations, New South Wales, 2000 to 2005

This evaluation determined that the surveillance system was performing a useful function, and was able to meet all of its pre-defined objectives to some extent. A principal limitation of the system was the inability to track the course of outbreaks efficiently and comprehensively, making central monitoring of the extent of outbreaks and the impact of control efforts difficult. In addition, the fact that enteric disease data was not linked with the NSW Notifiable Diseases Database delayed the sharing and reporting of outbreak information. A final limitation was the complicated, cumbersome and time-consuming nature of the enteric disease outbreak surveillance databases and information collation techniques, and the lack of flexibility of the data management system.

Given the absence of a system for tracking outbreaks and resultant policy outcomes, the findings presented here are likely to be incomplete. Nevertheless the system did appear to perform reasonably well with respect to prevention efforts and policy outputs. This report provides an imprecise indication of the comprehensiveness and timeliness of reporting due to the absence of a method for systematically tracking the course of an outbreak, and the absence of critical data fields, such as onset date of last case.

The sensitivity and specificity of the system could not be evaluated. Due to enhanced surveillance in the one site with the highest rates of reporting, the surveillance system is unlikely to be representative of the true distribution of enteric disease outbreaks. Furthermore, approximately 17.2 million cases of gastroenteritis occur in Australia annually, but only a minority of people with gastroenteritis go to a doctor, and only a minority of these provide a stool sample.⁷ Hence, the reported rates of enteric disease outbreaks are likely to be a substantial underrepresentation of the true rates of enteric disease outbreaks in the community. The rates of enteric outbreaks reported from this evaluation were comparable with previous estimates of reported rates of enteric disease outbreaks in NSW, indicating some system stability.⁸

The evaluation itself was limited: it was not independent of the health system, and only a limited number of key informants within the NSW Department of Health were interviewed. Local PHU staff were not interviewed, and may have been able to provide important insights into the performance and usefulness of the enteric disease surveillance system. This evaluation focussed mainly on the surveillance processes within the Department, and not on the notifications to PHUs, or the PHU investigations themselves. In addition, the cost-effectiveness of the system was not evaluated.

This evaluation produced some recommendations to improve the efficiency and effectiveness of the surveillance system. Recommendations from the review, together with the development of a tender for a new notifiable diseases surveillance database system, have resulted in agreement to the following improvements to the system:

• consolidation of all disease outbreak information into one database

• linkage of enteric disease outbreak data with enteric disease data in the redeveloped state and national notifiable disease databases

• development of a mechanism to monitor the course of outbreaks and assist in comprehensive outbreak data collection.

Once the new database has been constructed and implemented, simplification and consolidation of the data collection forms will be undertaken as a matter of priority.

The NSW enteric disease outbreak surveillance system is serving a useful public health function and should be continued. The system could be improved through the use of more sophisticated electronic data management techniques.