Are pregnancy outcomes associated with risk factor reporting in routinely collected perinatal data?

Population level data are well suited to studies evaluating health care. With the risk of sampling bias removed, estimation of incidence and prevalence rates can be made, allowing for description of the total burden of a particular disease or outcome, analysis of risk factors and trends, as well as identification of health inequalities and estimation of health costs.^1,2 Accurate conclusions from such analyses rely on high quality data that truly represent the population experience. Assessment of data quality (completeness and accuracy) is typically undertaken by a validation study, in which data from a sample of records from the population dataset are compared to a highly reliable and accurate source of data (‘the gold standard’) for the corresponding records. The accuracy and reliability of individual data items are typically reported.^3,4

The variables in perinatal population data can be continuous (e.g. gestational age), nominal (e.g. mode of delivery) and ordinal (e.g. first, second, third or fourth degree perineal tears), with validation of such variables typically reporting percent agreement and kappa statistics. These types of variables are frequently dichotomised in analyses (e.g. preterm birth, caesarean section, or third–fourth degree tears),^5,6 but little assessment has been undertaken into the accuracy and reliability of their dichotomised form.

Differential reporting in population health data occurs when a variable is reported with different accuracy and reliability amongst different strata of another variable. This can introduce systematic bias, leading to under or over-estimation of risk factor effects.⁷ For example, if smoking is more likely to be reported when an infant is growth restricted, this could result in the effect of smoking on growth restriction being over-estimated. Different accuracy and reliability statistics have been demonstrated for reporting of both pregnancy hypertension and induction depending on the mode of delivery,^2,8 and for hypertension depending on the gestation.⁹ However, we are only aware of one other study that has investigated whether the occurrence of adverse infant or maternal outcomes might result in increased reporting of established risk factors for these outcomes.⁹

With little published research reporting on the dichotomised form of population data, the aims of our study were therefore twofold: a) to assess reporting characteristics of commonly dichotomised pregnancy outcomes; and b) to investigate whether behaviours (e.g. smoking), medical conditions (e.g. diabetes) or interventions (e.g. induction) were reported differently by outcomes.

This study involved further analysis of data from a previous validation study of the 1998 New South Wales (NSW) Perinatal Data Collection (PDC). The PDC (formerly known as the NSW Midwives Data Collection) is a population-based statutory surveillance system and serves as a primary source of information about pregnancy and birth outcomes in NSW for all births ≥20 weeks gestation or ≥400 g birthweight. The original study is described in detail elsewhere.³ Briefly, randomly selected records from the PDC (referred to as the ‘PDC sample’) were compared with ‘gold standard’ data extracted from the corresponding patient’s medical records (referred to as the ‘validation data’). The PDC sample comprised 1680 records representing 2% of the state’s births from 98 hospitals around NSW. Information from the medical records of the selected sample of women was extracted by experienced health managers without reference to information contained in the PDC sample. The data item with highest frequency of missing values was Apgar5, which was missing from six records in the PDC sample (0.36%), and from nine records in the validation data (0.54%).

We first assessed the accuracy and reliability of continuous and polytomous data items when examined as dichotomous outcomes. We chose data items that are commonly dichotomised including: gestational age (<37 weeks gestation, ≥37 weeks gestation); birthweight (<2500 g, ≥2500 g; <4000 g, ≥4000 g); Apgar score at 1 minute (Apgar1 <4, Apgar1 ≥4) and Apgar score at 5 minutes (Apgar5 <7, Apgar5 ≥7); epidural, caudal, pudendal or spinal analgesia (regional analgesia yes/no); second, third or fourth degree tears and/or episiotomy (perineal trauma yes/no); and baby discharge status (stillbirth/live birth).

Next we examined potential differential reporting of risk factors by determining the accuracy and reliability of risk factor reporting in the PDC sample for different pregnancy outcomes. Specifically, we hypothesised that the following established risk factors may be more likely to be reported in the presence of an associated outcome:

• smoking when infants were small or preterm¹⁰

• maternal hypertension among preterm births¹¹

• maternal diabetes when infants were large¹²

• instrumental birth (forceps or vacuum) among women who experienced perineal trauma¹³

• induction among women who required regional analgesia¹⁴

• infant resuscitation (intermittent positive pressure respiration, bag and mask or intubation, or external cardiac massage and ventilation) when Apgar5 <7.

Using the validation data as the ‘gold standard’, the reliability and accuracy of PDC reporting was determined by calculating the sensitivity, specificity, positive predictive value (PPV), negative predictive value, percent agreement and Cohen’s kappa statistic. These reporting characteristics were determined first for the commonly dichotomised variables and then for risk factors in the hypothesised outcome strata. When a record was missing a data item, it was excluded from the relevant analysis. We assessed the homogeneity of risk factor reporting across the dichotomised outcome strata by the Breslow-Day test, with Zelan adjustment where cell counts were less than five.

All analyses included the associated 95% exact binomial confidence intervals. These are not presented in the tables, but are available from the authors on request. All analyses were undertaken using SAS (version 9.2, SAS Institute, Cary, NC, USA).

Of the 1680 records in the original validation study, 1678 were available for analysis. Characteristics of the PDC sample were representative of all births in NSW (Table 1).

Table 1.  Comparison of Perinatal Data Collection (PDC) sample with all NSW births, 1998

Commonly dichotomised pregnancy outcomes (preterm birth, low and high birthweight, Apgar scores, perineal trauma, regional analgesia and stillbirth) as reported in the PDC had excellent levels of agreement, and high levels of ascertainment (sensitivities >94%) and accuracy (PPVs >96%) (Table 2).

Table 2.  Agreement, ascertainment and accuracy of dichotomised pregnancy outcome variables reported in the Perinatal Data Collection (PDC) compared with validated data, NSW, 1998

The results of the investigation into differential reporting are presented in Table 3. PPVs were high, with 11 of 14 individual analyses ≥90%, but with inconsistencies in direction among outcome groups for each risk factor. There was more variability in the sensitivities, ranging from 66% for reporting of infant resuscitation amongst the group whose Apgar5 was ≥7, to 99% for reporting of inductions with no regional analgesia. In total, six out of the 14 sensitivity measures were ≥90%. There was no overall pattern suggestive of better reporting in the presence of an adverse outcome. Although there was a trend to higher ascertainment of infant resuscitation among infants with low Apgar5 (sensitivities of 86% vs 66%), of instrumental birth among women with perineal trauma (97% vs 88%), and of hypertension among preterm birth (77% vs 67%), the reverse was true for ascertainment of smoking both among preterm birth (82% vs 90%) and among small infants <2500 g (83% vs 90%). There were no statistically significant differences in reporting across strata, with Breslow-Day p values all >0.05.

Table 3.  Agreement, ascertainment and accuracy of dichotomised pregnancy risk factors reported in the Perinatal Data Collection (PDC) and grouped by pregnancy outcomes compared with validated data, NSW, 1998

This study demonstrated that dichotomising perinatal outcome data into categories that are typically reported in population health research^5,6 resulted in high levels of ascertainment and accuracy. With all sensitivities ≥94.7% and all PPVs ≥96.1%, reassurance is provided for the use of these data items in their dichotomised form where necessary for comparison to other findings or due to sample size constraints. There was no evidence of overall systematic bias in risk factor reporting across one strata of outcome (the adverse group) compared to the other. This study adds new information on dichotomised reporting characteristics and differential reporting. Strengths of this study include the highly representative nature of the PDC sample, the use of six measures of accuracy and reliability, and the small percentage of missing data. Limitations included small numbers in some outcome strata. Lack of statistical significance may thus have been a result of underpowering for some categories.

Most risk factors were fairly well ascertained regardless of outcome strata, with the exception of hypertension and infant resuscitation among the groups that did not have an adverse outcome. Reliability, as measured by PPV, was lowest amongst diabetes reporting for the adverse group, but numbers were small. There was a non-significant trend towards higher ascertainment of hypertension, instrumental birth and infant resuscitation in the adverse groups. It is recognised that these trends could become significant with larger sample sizes, and may introduce biases in research.

The non-significant trends in differential reporting were not always in the hypothesised direction. Ascertainment for behaviour (smoking) was lower amongst the adverse outcome group, while ascertainment for some interventions (instrumental birth and infant resuscitation) and for hypertension was higher in the adverse outcome groups. This latter finding is consistent with another study that identified a trend towards increased ascertainment of hypertension among women who delivered prematurely or suffered a morbidity.⁹ While it might be expected that some risk factors which may be reported earlier in pregnancy (e.g. smoking, hypertension) may not have the same impact on reporting as risks occurring closer to delivery (e.g. induction, infant resuscitation), there were no differences in ascertainment or accuracy for these factors. Overall our findings demonstrate the randomness of reporting errors and no evidence of systematic bias due to differential reporting by outcome.

This study used data collected in 1998 as this was the last time the PDC was validated against medical records. Some changes to the recording of information are likely to have occurred with the advent of electronic systems, but the majority of PDC recording still occurs at the time of the birth admission, and hence accuracy of variables once dichotomised and of maternal or infant outcome risk factor reporting are unlikely to have been affected.

Our findings demonstrate that dichotomised perinatal variables have high levels of accuracy and reliability when compared with medical records. In addition, ascertainment of risk factors show some non-significant differences within different pregnancy outcome groups; however reporting errors are random in their direction, revealing that there is no evidence of systematic bias.