External quality control processes for infectious disease testing

Introduction

Historically, serological testing for infectious diseases was performed using biological assays such as complement fixation or haemagglutination inhibition. These assays utilised the agglutination or haemolysis of red blood cells as biological indicators for the presence or absence of antibodies.¹ Generally, a four-fold difference in doubling dilution titres was required to consider a significant difference in antibody levels. Over the 1990s, with the advent of enzyme immunoassays (EIAs), testing for infectious disease antibodies moved away from biological assays, which were labour intensive and difficult to control, to microtiter plate EIAs and then to automated platforms.^1,2 The output of these tests is reported in a unit of measure calculated from the intensity of the signal produced by the reaction, be it colorimetric, immunofluorescent or chemiluminescent. This signal, often referred to as a signal to cut-off (S/Co) is an arbitrary unit based on the comparison of the signal produced by the patient sample compared with a cut-off determined by the manufacturer e.g. a multiplier of the negative control signal or the mean value of particular calibrators. The ‘cut-off’ value for these assays effectively becomes the assay decision point, separating the populations of samples with the target analyte from those that do not contain the target analyte. Whereas the S/Co or other arbitrary unit will generally increase as the amounts of antibody increases in sample tested, the test system is not measuring the quantity of antibodies present, it measures the amount of binding of the antibody in the patient sample with the antigen on the solid phase of the assay.^1–3

Testing for infectious diseases using immunoassays gradually became available on high-throughput immunoassay platforms that also test for clinical chemistry markers. In more advanced countries, infectious disease testing has moved away from the microbiology laboratory into ‘core laboratories’, where the instruments and associated processes, including the quality control processes, are managed using a singular system within the same laboratory, typically the traditional approaches applied to clinical chemistry testing. However, testing for inert chemicals such as glucose and potassium measure the amount of analyte in the patient sample. In these situations, the test system is calibrated to a standard, often an international standard, and the results are expressed in SI units. This lends itself to certain statistical methodologies. By contrast, the arbitrary S/Co result obtained in infectious disease testing is influenced by a range of factors relating to the antibodies being detected including their avidity or affinity, genotype or subtype of causative agent, stage of disease progression, immune status of the patient, and factors relating to the assay itself such as target antigens, antibodies utilised in the conjugate, and chemistry applied to create and detect the signal.⁴

Quality control

The use of a quality control (QC) sample is a requirement for laboratories accredited to ISO 15189 and is defined in the standard as an ‘internal procedure which monitors the testing process to decide if the system is working correctly and gives confidence that the results are reliable enough to be released’ (section 3.11, p. 3⁵). Further in the ISO 15189 document it states, ‘The procedure should also allow for the detection of lot-to-lot reagent and/or calibrator variation of the examination method.’ (section 7.2.7.2(a), p. 25⁵). The National Association of Testing Authorities (NATA) ISO 15189 Standard Application Document (SAD) refers to QC processes as ‘A system must be established for the long-term monitoring of internal quality control results to assess method performance’ (section 5.6.2, p. 11⁶). Frequently, laboratories will interpret the standard to suggest that the use of a kit control is adequate to fulfil the requirement.

Kit controls

Kit controls in infectious disease testing have the purpose of validating the test. Generally, kit controls are tested, and the results are accepted prior to testing patient samples. The manufacturer provides the kit controls and associated acceptance criteria. These acceptance criteria have been developed by the manufacturer in pre-market clinical trials and results within the established range can be taken as evidence that the test kit is performing as expected by the manufacturer and the sensitivity and specificity claimed by the manufacturer can be assured. It is often pointed out that the acceptance range for kit controls are wide. This is in fact the case because infectious disease serology assays tolerate significant changes in signal before the clinical sensitivity and specificity is compromised. Note that historically biological assays allowed a four-fold change in dilutions before a significant difference was confirmed. Kit controls are required to be tested when stated in the manufacturer’s instructions for use (IFU). All infectious disease assays are listed on the Australian Registry for Therapeutic Goods (ARTG) as class 3 or class 4 in vitro diagnostic devices (IVD).⁷ Laboratories reporting clinical results are required to follow the IFU without deviation. Any modification to the IFU, such as not using a specified kit control, means the assay is being used ‘off licence’ and becomes an ‘in-house IVD’ which must be registered as such with the Therapeutic Goods Administration (TGA). In cases where the manufacturer’s IFU does not require the testing of the kit control, their use is highly recommended as best practice. Kit controls should not be replaced with third party controls but utilised in conjunction with them.

Third party controls

The ISO 15189 standard states ‘To enable this, (the ability to detect lot to lot variation) the use of third-party IQC material should be considered, either as an alternative to, or in addition to, control material supplied by the reagent or instrument manufacturer’ (section 7.2.7.2(a), p. 25⁵). Whereas the kit controls are designed to validate the assay at the time of testing, they are not designed to monitor the performance of the assay over time. Generally, kit controls are not sensitive to changes in the test system. Well-designed third-party controls are IVDs manufactured by companies other than the test kit manufacturer and are designed to monitor variation.^4,8 These controls should have a reactivity at a level that can detect variation. The ISO 15189 standard states ‘the IQC material provides a clinically relevant challenge to the examination method, has concentrations levels at or near clinical decision points’ (section 7.2.7.2(b), p. 26⁵). Immunoassays do not have a linear dose response curve. That is, as the amount of analyte being detected increases, it is expected that the signal will increase proportionally. In most immunoassays, the dose response curve is sigmoidal. Initially as the amount of analyte increases there is only a small increase in signal. As the analyte concentration increases, the curve becomes linear until such time that all or part of the components are exhausted, after which the curve plateaus. The third-party controls must therefore be reactive in the linear part of the curve to be effective in detecting variation, and the linear part of a curve may not necessarily be close to the cut-off of an assay.¹

The NATA SAD states, ‘Numerical QC results should be presented graphically to assist in the early detection of trends’ (section 5.6.2, p. 11⁶). Infectious disease testing has a numerical value (the S/Co or other arbitrary unit), noting that these numbers are not a measure of an amount of antibodies, rather a measure of binding activity. However, they can be plotted on a Levey-Jennings chart and effectively monitor variation in the test system. If the supplier of the third-party control has minimal lot-to-lot variation, the results of multiple lots of the same third-party QC can be used to monitor the assay over many years, providing the laboratory good insight into the assays long-term precision and bias.⁹

The use of a third-party QC optimised for the assay being monitored is highly encouraged. It serves a different, but complimentary, purpose to the kit controls and laboratories should use both the kit control and the third-party control. At a minimum the use of either the kit control or third-party control is mandatory for laboratories accredited to ISO 15189.⁵

Acceptance ranges for third party controls

Guidance on how QC results are managed is limited. The NATA SAD states, ‘A system must be established for the long-term monitoring of internal quality control results to assess method performance’ (section 5.6.2, p. 11⁶). The National Pathology Accreditation Advisory Committee (NPAAC) Requirements for Quality Control, External Quality Assurance and Method Evaluation document, quality control section states ‘For quantitative assays, target values and SDs must be determined using laboratory data’ but does not specify how these ranges are determined.¹⁰ Traditionally, clinical chemists have used the mean ± x standard deviation (s.d.) of a small data set (e.g. 20–30 results) and applied Westgard rules to identify unexpected variation.^11,12 As infectious disease testing moved from the microbiology laboratory to the ‘core laboratory’, it is unsurprising that these traditional methods have been applied to the S/Co or arbitrary values expressed by the immunoassays. However, it has long been anecdotally recognised, and recently published, that infectious disease testing experiences significant reagent lot-to-lot variation and the use of traditional QC methods causes unacceptable numbers of false rejections.^4,13 Therefore, when an acceptance range based on 20–30 QC results is used to establish the QC acceptance range, frequently new reagent lots cause the QC to be out of range and therefore rejected. The laboratory is therefore faced with a dilemma. Do they reject the reagent based on the QC result, noting that the kit controls are usually within the manufacturer’s acceptance criteria, indicating no change in sensitivity or specificity? Or do they re-calculate the range using the next 20–30 results, with the knowledge that the introduction of a new reagent lot will repeat the same situation? The laboratory would also need to justify why it is appropriate to release patient results using multiple acceptance criteria over time.

It should be noted that recalculating the mean and s.d. on a new reagent lot only re-establishes the imprecision of the assay. However, the change in reactivity of the QC is not due to a change in imprecision, but an introduction of bias caused by the new reagent. Recalculation of the mean and s.d. therefore ignores the root cause of the change and does not address the fundamental question of how much variation due to reagent lot change is acceptable.

Irrespective of the method utilised to establish the acceptance range for each QC sample, the methodology must be based on scientific evidence using data from the same test process being controlled, rather than assuming commutability of methodology. As the ISO 15198 standard states, when selecting a QC methodology, ‘The intended clinical application of the examination should be considered, as the performance specifications for the same measurand may differ in different clinical settings’ (section 7.2.7.2(a), p. 11⁵). This evidence should be made available to an auditor when requested.

Infectious disease specific QC requirements

The ISO 15189 standard does not specify which quality control methodologies should be employed.⁵ Like most standards designed for a broad set of disciplines, it is not prescriptive. This is also the case for the NATA SAD and NPAAC Requirements for Quality Control, External Quality Assurance and Method Evaluation documents, unlike the UK-equivalent Standards for Microbiology Investigations, Quality Assurance in the Diagnostic Infection Sciences Laboratory document, which implies the use of traditional methods including Westgard rules for use in infectious disease serology.¹⁴ This UK standard, however, has been recently modified to acknowledge that traditional methods are not perfect and thus includes alternative methods of QC including the use of QConnect limits.

The NATA SAD does provide additional discipline-specific QC requirements, including cartridge-based assays, chemical pathology, cytology, haematology and histopathology. To address the points relating to the provision of QC methods for infectious disease testing raised above, an additional infectious disease disciple-specific section will be added to the NATA SAD. These clauses will be included in the accreditation of medical testing laboratories in Australia.

The inclusion states:

Disclosure statement

Belinda McEwan in the ASM-authorised representative on the National Association of Testing Authorities and Human Pathology Accreditation Advisory Committee of the National Pathology Accreditation Advisory Council.