Evaluation of between-assay consistency among laboratory testing methods for neurosyphilis: a systematic review
Xu Zhang![https://orcid.org/0000-0002-9373-5211](/media/client/orcid_16x16.png)
![https://orcid.org/0000-0003-2788-5438](/media/client/orcid_16x16.png)
![https://orcid.org/0000-0003-0654-3826](/media/client/orcid_16x16.png)
![https://orcid.org/0000-0001-7364-4649](/media/client/orcid_16x16.png)
A Institute of Dermatology, Chinese Academy of Medical Science & Peking Union Medical College, National Center for STD Control, China Centers for Disease Control and Prevention, Nanjing 210042, China.
B Department of Dermatology, The Second Affiliated Hospital of Nanjing Medical University, Nanjing 210011, China.
Abstract
We conducted a systematic review to analyse the consistency of nontreponemal-specific tests of Treponema pallidum in cerebrospinal fluid. We searched the PubMed, EMBASE, Web of Science, CNKI, Wanfang and Chongqing VIP databases. The inclusion criteria were studies conducted on nontreponemal-specific tests in cerebrospinal fluid (CSF) within the same population. Exclusion criteria were studies with incomplete data or where we were unable to obtain the full text, duplicate reports, case reports and studies without sensitivity or specificity results. We used kappa value analysis and McNemar’s test to analyse study consistency. We initially collected a total of 198 articles and ultimately included six articles that involved 429 patients with neurosyphilis. The performance between venereal disease research laboratory tests (VDRL) and the reactive plasma regain or toluidine red serum unheated test was similar. The kappa value for consistency between VDRL and reactive plasma regain was >0.8 in three articles, and was 0.892 for consistency between VDRL and toluidine red serum unheated test in one article. Our results suggested that CSF-reactive plasma regain or CSF-toluidine red serum unheated test may serve as alternative tests in the diagnosis of neurosyphilis with CSF-VDRL.
Keywords: consistency, CSF, diagnosis, neurosyphilis, review, RPR, TRUST, VDRL.
Introduction
Neurosyphilis is an infectious disease caused by Treponema pallidum, a spirochaete bacterium that invades the central nervous system. Neurosyphilis can occur at any stage of syphilis, and it can cause serious neurological damage or even death due to delayed diagnosis or belated treatment.1 The clearly recommended laboratory diagnostic indicators that include the venereal disease research laboratory (VDRL) test, fluorescent treponemal antibody absorption (FTA-ABS) test, white blood cell count and protein quantification in cerebrospinal fluid (CSF) are in accordance with the most recent version of China’s ‘Sexually Transmitted Diseases Clinical Diagnosis, Treatment and Prevention Guidelines’ and the US Centers for Disease Control and Prevention’s guidelines for sexually transmitted disease diagnosis and treatment.2,3 Of these tests, CSF-VDRL is recommended as one of the laboratory diagnostic indicators for neurosyphilis due to its high specificity.4 However, the laboratory conditions required to operate the CSF-VDRL test are of relatively high complexity, as the reagent needs to be prepared and must be used within 2 h, and a light microscope is required for detection, making it difficult for many laboratories to execute the test effectively. Therefore, the reactive plasma regain (RPR) test and the toluidine red serum unheated test (TRUST) may be reasonable alternative choices. RPR and TRUST are commonly used to detect non-T. pallidum-specific antibodies in peripheral blood, and they carry relatively high specificity and sensitivity.5 However, it is unknown whether the specificity and sensitivity remain high when detecting non-T. pallidum-specific antibodies in CSF or if they constitute appropriate alternatives to VDRL. Therefore, a number of controlled clinical studies have been conducted that focus on the differential sensitivity and detection rates of neurosyphilis between VDRL and RPR/TRUST in CSF; but these have produced inconsistent conclusions, and controversies persist that require resolution. We therefore searched studies over the past 30 years in which comparisons were made between VDRL and RPR or with TRUST in CSF, and we exploited kappa consistency analysis and McNemar’s chi-squared test to determine the consistency of the conclusions derived from various studies. We posit that CSF-RPR and CSF-TRUST can be systematically evaluated as to their feasibility as alternative tests to CSF-VDRL.
Methods
Data sources and searches
We searched PubMed, Embase, Web of Science, China CNKI, China Wanfang and China VIP databases between January 1990 and June 2021. We used the following search terms: (1) ‘CSF’ or ‘Cerebrospinal Fluid’ and ‘neurosyphilis’ and ‘RPR’ and ‘VDRL,’ and (2) ‘CSF’ or ‘Cerebrospinal Fluid’ and ‘neurosyphilis’ and ‘TRUST’ and ‘VDRL’.
Inclusion and exclusion criteria
Our inclusion criteria were as follows: (1) a diagnosis of definite or probable neurosyphilis, (2) studies in which the performance of different tests were evaluated among the same group of patients, (3) studies with clear results regarding the sensitivity and specificity of various tests, and (4) articles published in English or Chinese only. Our exclusion criteria were as follows: (1) articles without complete information or a full text, (2) repeated reports, (3) case reports, and (4) review articles.
Study selection
The selection and quality evaluation were separately conducted by two authors. Full texts were obtained for any potentially relevant articles and reviewed to determine whether they met our inclusion criteria. If disagreements occurred over the choice of an article, two authors worked to reach a consensus through discussion or asked for advice from a third author. The useful information that we incorporated included Author, Publication Date, Periodical Name, Study Date, Diagnostic Criteria for Neurosyphilis, Number of Patients, HIV Status, and Sensitivity and Specificity results.
Statistical analysis
We herein reported the sensitivity and specificity of different CSF tests for neurosyphilis and used kappa value analysis and McNemar’s chi-squared test to analyse the consistency between different pairs of tests. All statistical analyses were conducted in R 3.5.2 (The R Foundation for Statistical Computing).
Results
Included articles
In our search, we identified 198 articles after removing 55 duplications. We then made a further elimination under the following orders: 25 were non-neurosyphilis studies, 123 were case reports, 20 conducted different tests among different groups of patients, 15 did not show sensitivity and specificity results, and nine did not provide diagnostic criteria. As a result, six met the inclusion criteria of the present study (Fig. 1).6–11 These studies encompassed 429 patients who were classified as exhibiting definite neurosyphilis, 49 as having probable neurosyphilis and 1668 as not having neurosyphilis (Tables 1 and 2).
Study | Criteria for definite neurosyphilis | Criteria for probable neurosyphilis | Cases of definite neurosyphilis | Cases of probable neurosyphilis | Cases without neurosyphilis | Cases with HIV co-Infection | Sensitivity by VDRL (%) | Specificity by VDRL (%) | Sensitivity by RPR (%) | Specificity by RPR (%) | |
---|---|---|---|---|---|---|---|---|---|---|---|
Versiani et al. 6 | Clinical suspicion and CSF-VDRL+ | Clinical suspicion | 21 | 49 | 50 | U | 100 | 100 | 100 | 100 | |
Zhu et al. 7 | (1) CSF-TPPA+ and clinical suspicion (2) CSF-TPPA+ and WBC ≥10/μL | NA | 210 | NA | 922 | N | 81.4 | 90.3 | 76.2 | 93.4 | |
Castro et al. 8 | (1) RPR ≥1:8, MHA-TP ≥1:80 and CSF-FTA-ABS+ (2) CSF WBC ≥10/μL or CSF protein ≥45 mg/dL | NA | 24 | NA | 290 | Y | 99 | 70.8 | 99.3 | 75 | |
Marra et al. 9 | CSF-FTA-ABS+ and CSF WBC >20/μL | NA | 72 | NA | 77 | Y | 71.8/66.7A | 98.3/80.2 | 56.4/51.5 | 100/89.7 | |
Lin et al. 10 | CSF-VDRL+ | NA | 61 | NA | 295 | U | 100 | 100 | 93.4 | 99.3 |
CSF, cerebrospinal fluid; RPR, rapid plasma reagin; VDRL, venereal disease research laboratory test; TPPA, Treponema pallidum particle agglutination; WBC, white blood cells; MHA-TP, microhaemagglutination assay for Treponema pallidum; U, unknown; N, no; NA, not applicable; Y, yes.
ALaboratory-defined neurosyphilis/symptomatic neurosyphilis.
Study | Criteria for definite neurosyphilis | Criteria for probable neurosyphilis | Cases of definite neurosyphilis | Cases of probable neurosyphilis | Cases without neurosyphilis | Cases with HIV co-infection | Sensitivity by VDRL (%) | Specificity by VDRL (%) | Sensitivity by TRUST (%) | Specificity by TRUST (%) | |
---|---|---|---|---|---|---|---|---|---|---|---|
Zhu et al. 7 | (1) CSF-TPPA+ and clinical suspicion (2) CSF-TPPA+ and WBC ≥10/μL | NA | 210 | NA | 922 | N | 81.4 | 90.3 | 76.2 | 93.1 | |
Jiang et al. 11 | (1) CSF-VDRL+ (2) CSF WBC >5/μL and CSF-TPPA+ | NA | 41 | NA | 34 | N | 93.1 | 100 | 94.7 | 100 |
CSF, cerebrospinal fluid; TRUST, toluidine red unheated serum test; TPPA, Treponema pallidum particle agglutination; VDRL, venereal disease research laboratory test; WBC, white blood cells; N, no; NA, not applicable.
Consistency evaluation between CSF-VDRL and CSF-RPR
Five of the six studies entailed a comparison between CSF-VDRL and CSF-RPR as to their sensitivity and specificity. The sensitivity of CSF-VDRL was between 66.7% and 100%, and the specificity ranged from 70.8% to 100%, whereas the sensitivity of RPR ranged from 51.5% to 100%, and the specificity ranged from 75% to 100% (Table 1). We performed a kappa value analysis and found that the consistency was high (kappa > 0.8) in three studies; two of them had the same conclusion under McNemar’s chi-squared test (P > 0.05; Supplementary Table S1A). The remaining two studies revealed a relatively lower consistency (kappa = 0.717 and 0.560), and McNemar’s chi-squared test supported the result in one of them (P < 0.05; Table S1A).
Consistency evaluation between CSF-VDRL and CSF-TRUST
In two of the six studies, the authors compared CSF-VDRL and CSF-TRUST. The sensitivity with respect to CSF-VDRL was between 81.4% and 93.1%, with specificity from 90.3% to 100%; the sensitivity of TRUST was between 76.2% and 94.7%, with a specificity of 93.1% to 100% (Table 2). Based on data from one article, we determined high consistency for these two tests (kappa = 0.892; Table S1B).
Influence of clinical symptoms on consistency evaluation
The sensitivity of CSF-VDRL and CSF-RPR was 66.7–85.7% versus 51.5–81.5% in symptomatic neurosyphilis patients, and 69.6–71.8% versus 56.4–60.7% in asymptomatic patients; the specificity was 80.2–86.7% versus 89.7–90.2% in the former, and 79.4–98.3% versus 82.6–100% in the latter patients (Table S2A). One study also showed that the sensitivity of CSF-VDRL and CSF-TRUST was 85.7% versus 82.5% in the former group, and 69.6% versus 58.9% in the latter group, whereas the specificity was 86.7% veresus 90.1% in symptomatic patients, and 79.4% versus 82.1% in asymptomatic patients (Table S2B).
Influence of HIV status on consistency evaluation
In one study with only HIV-negative patients, the investigators demonstrated that the sensitivity of CSF-VDRL and CSF-RPR was 81.4% versus 76.2%, and that the specificity was 90.3% versus 93.4%, which led to an extremely high consistency coefficient of 0.884 (Table S3). In addition, two studies that included HIV-positive patients reflected a sensitivity of CSF-VDRL and CSF-RPR of 66.7–99.0% versus 51.5–99.3%, respectively, and a corresponding specificity of 70.8–98.3% versus 75–100% (Table S3).
Discussion
The CSF-VDRL has a high specificity and is currently considered the definitive test for the diagnosis of neurosyphilis globally. However, the antigen used in VDRL is combined with proportional cardiolipin, lecithin and cholesterol, and without any observable trace flags. Thus, the CSF-VDRL relies on a light microscope and skilled technicians, which made it difficult to popularise. In this review, we comprehensively analysed the consistency between CSF-VDRL and CSF-RPR, or CSF-TRUST, after collecting the data from previous studies. Our results indicated that CSF-RPR and CSF-TRUST were highly consistent with CSF-VDRL, showing that they may constitute substitute methods for the laboratory diagnosis of neurosyphilis.
Both RPR and VDRL are non-treponemal tests with similar experimental principles. RPR is posited to be an appropriate test for serum and plasma, as it uses charcoal particles as the stabiliser, and RPR can also be used today to detect antibodies against cardiolipin antigen (among other bodily fluids, such as CSF) due to technological advancements.4 Compared with VDRL, the RPR test can be conducted in laboratories and in less-developed areas with relatively lower requirements for experimental sophistication. There is still no definite conclusion whether CSF-RPR can substitute for CSF-VDRL, although several studies have uncovered a fairly similar performance between them.6–8 Our study showed high consistency between CSF-RPR and CSF-VDRL in neurosyphilis diagnosis, supporting the view that CSF-RPR can provide an alternative choice to VDRL. According to this conclusion, the diagnostic efficiency of neurosyphilis is expected to improve significantly, particularly in areas where VDRL cannot be executed.
The TRUST method is a modified version of VDRL, and as the toluidine red particle-labelled antigen is stable at room temperature, the TRUST reagents have been successfully applied in China and in many other countries.7,11 Similar to the case for RPR, TRUST can also detect antibodies against the cardiolipin antigen in CSF due to its technological improvements. In one study, the authors ascertained a 97.3% consistency between CSF-TRUST and CSF-VDRL after using McNemar’s chi-squared test (Gu et al.);12 others found a kappa value of 0.709 between CSF-VDRL and other nontreponemal tests in CSF among symptomatic neurosyphilis patients, and a kappa value of 0.692 between CSF-VDRL and CSF-RPR or CSF-TRUST among asymptomatic patients (Zhu et al.).7 These results are in good agreement with our study, suggesting that CSF-TRUST and CSF-VDRL are highly consonant. Thus, the CSF-TRUST can also be employed in many geographic areas as an alternative test in neurosyphilis diagnosis. It is worth noting that the CSF-VDRL instructions call for diluting the VDRL antigen and using less antigen than in the serum version to account for lower antibody concentrations in CSF compared with serum, to avoid the prozone phenomenon and reduce false negative test results. Thus, if researchers can perform the RPR or TRUST on CSF using the method recommended for the CSF-VDRL, it may decrease the number of false negative results and further increase the consistency.
In our study, we also focused on other factors that might affect consistency between different types of tests. As a result, only a tiny difference was observed in kappa values (CSF-RPR vs CSF-VDRL) between HIV-positive and HIV-negative patients, suggesting that HIV infection may not influence the test comparisons. We additionally confirmed that symptoms may play an important role in the performance of different tests, as laboratorial physicochemical indices were originally different between symptomatic and asymptomatic patients – and this may then further explain our findings and remind researchers to consider this effect in future comparison studies.
We compare different CSF tests from the point of view of consistency analysis, which may lead to selection bias compared with meta-analysis. Another limitation is that the individual-level data were unavailable from these collected articles, which prevented us from further analysis. Furthermore, the final consistency analysis included a relatively small dataset due to the fact that only a portion of the analyses encompassed the complete datasets needed for calculation. Further, the literature we collected may not have provided enough comprehensive information on patients, such as symptoms and HIV status, which may have influenced testing performance to a degree. Finally, the studies included in the review used different definitions for neurosyphilis, and may affect test performance among different types of neurosyphilis, such as symptomatic and asymptomatic neurosyphilis.
To summarise, we have herein suggested that CSF-RPR or CSF-TRUST were consistent with CSF-VDRL, which would make it possible to substitute VDRL as the neurosyphilis diagnostic method of choice. Popularisation of CSF-RPR and CSF-TRUST may, therefore, facilitate the improvement in diagnostic ability regarding neurosyphilis, especially in those geographic areas where CSF-VDRL is hard to implement. It is worth noting that CSF-VDRL is still acknowledged as an imperfect test for neurosyphilis diagnosis, although it is the most recommended test nowadays. Thus, the best solution in the future is to evaluate novel tests that may possess superior performance and provide less difficulty in producing a diagnosis of neurosyphilis.
Acknowledgements
The authors thank all the authors who contributed to articles that were included in this review.
References
1 Ropper AH. Neurosyphilis. N Engl J Med 2019; 381(14): 1358-63.
| Crossref | Google Scholar |
2 National Center for STD Control, Chinese Center for Disease Control and Prevention; Venereology Group, Chinese Society of Dermatology; Subcommittee on Venereology, China Dermatologist Association. Guidelines for diagnosis and treatment of syphilis, gonorrhea and genital chlamydia trachomatis infections (2020). Chin J Dermatol 2020; 53: 168-79.
| Crossref | Google Scholar |
3 Workowski KA, Bachmann LH, Chan PA, Johnston CM, Muzny CA, Park I, et al. Sexually transmitted infections treatment guidelines, 2021. MMWR Recomm Rep 2021; 70(4): 1-187.
| Crossref | Google Scholar |
4 Larsen SA, Steiner BM, Rudolph AH. Laboratory diagnosis and interpretation of tests for syphilis. Clin Microbiol Rev 1995; 8(1): 1-21.
| Crossref | Google Scholar |
5 Satyaputra F, Hendry S, Braddick M, Sivabalan P, Norton R. The laboratory diagnosis of syphilis. J Clin Microbiol 2021; 59(10): e0010021.
| Crossref | Google Scholar |
6 Versiani I, Cabral-Castro MJ, Puccioni-Sohler M. A comparison of nontreponemal tests in cerebrospinal fluid for neurosyphilis diagnosis: equivalent detection of specific antibodies. Arq Neuropsiquiatr 2019; 77(2): 91-5.
| Crossref | Google Scholar |
7 Zhu L, Gu X, Peng RR, Wang C, Gao Z, Zhou P, et al. Comparison of the cerebrospinal fluid (CSF) toluidine red unheated serum test and the CSF rapid plasma reagin test with the CSF venereal disease research laboratory test for diagnosis of neurosyphilis among HIV-negative syphilis patients in China. J Clin Microbiol 2014; 52(3): 736-40.
| Crossref | Google Scholar |
8 Castro R, Prieto ES, da Luz Martins Pereira F. Nontreponemal tests in the diagnosis of neurosyphilis: an evaluation of the Venereal Disease Research Laboratory (VDRL) and the Rapid Plasma Reagin (RPR) tests. J Clin Lab Anal 2008; 22(4): 257-61.
| Crossref | Google Scholar |
9 Marra CM, Tantalo LC, Maxwell CL, Ho EL, Sahi SK, Jones T. The rapid plasma reagin test cannot replace the venereal disease research laboratory test for neurosyphilis diagnosis. Sex Transm Dis 2012; 39(6): 453-7.
| Crossref | Google Scholar |
10 Lin L, Yang R, Zhang X, Xu L, Song W, Bi C, et al. Comparisons of several laboratory tests in the diagnosis of neurosyphilis. Chin J Dermatol 2011; 44(2): 127-9.
| Google Scholar |
11 Jiang Y, Chen X, Ma X, Yang Y, Peng F, Hu X. The usefulness of toluidine red unheated serum test in the diagnosis of HIV-negative neurosyphilis. Sex Transm Dis 2011; 38(3): 244-5.
| Crossref | Google Scholar |
12 Gu W, Yang Y, Wu L, Yang S, Ng L-K. Comparing the performance characteristics of CSF-TRUST and CSF-VDRL for syphilis: a cross-sectional study. BMJ Open 2013; 3(2): e002204.
| Crossref | Google Scholar |