Commentary: lessons from molecular genetic studies on reporting false-positive results

Keywords: gene expression, genetics, genotyping, reproduction.

Poor replication of published research results is a common problem. These issues were highlighted in the publication ‘Why Most Published Research Findings Are False’ by John Ioannidis (2005), leading to considerable debate. Major contributors to the problem included pressure to publish, inadequate study design and bias towards publishing positive results. Small sample sizes, inappropriate data analysis and poor statistical inference also contributed.

Failure to adequately control for multiple testing is a common problem of statistical inference leading to publication of false-positive results. In this regard, the evolution of mapping studies to identify genetic risk factors for common diseases has been instructive. In the past 10 years, genomic locations for thousands of genetic risk factors for common diseases have been mapped through genome-wide association studies (GWAS; Visscher et al. 2017). Empirical results for well-replicated genetic risk factors identified by GWAS demonstrate that effect sizes are small and the significant genetic variants are mostly located within introns and intergenic regions, not in coding regions of genes. Large GWAS studies reliably detect these small effects and the results replicate well across studies because, at the outset, the field set a genome-wide threshold for significance (P < 5 × 10⁻⁸) to account for testing association with variants from across the genome (The International HapMap Consortium 2005; Pe’er et al. 2008; Fadista et al. 2016).

GWAS superseded candidate gene studies from the previous decade. How well do those candidate gene studies replicate in GWAS results? Not well: across multiple diseases and with only a handful of exceptions, GWAS results have shown that published candidate gene associations were mostly false-positive results (Montgomery et al. 2008; Duncan et al. 2019). The candidate gene studies generally used small samples and were underpowered for the detection of the small size effects now reliably detected in the much larger samples used for GWAS. One reason for the failure is that common variants with the large genetic effects that may have been reliably detected with the small samples used in candidate gene studies are rare for complex diseases.

In addition, there was a failure to appreciate and adequately account for the multiple testing issues. Choosing known variants in coding regions of biologically plausible genes turned out not to increase the chances of finding true results, suggesting we knew less about the biology than we thought (Chabris et al. 2012; Rahmioglu et al. 2012; Duncan et al. 2019). Genetic variants in coding regions of candidate genes were no more likely to be associated with risk for common complex diseases than variants elsewhere in the genome. Consequently, the appropriate distribution to apply in multiple testing correction should have been the genome-wide level of significance used as the standard for reporting results from GWAS. Worse still, many studies failed to correct within their own studies for testing more than one variant or testing multiple genetic models, and results were declared significant at nominal values of P < 0.05. Thresholds of P < 0.05 or P < 5 × 10⁻⁸ give very different estimates of the number of ‘significant’ hits. For example, a GWAS meta-analysis on endometriosis with 17 045 endometriosis cases and 191 596 controls identified 14 independent risk variants at a genome-wide level of significance (Sapkota et al. 2017). In contrast, 24 431 variants passed the nominal significance threshold of P < 0.05.

Underpowered candidate gene association studies continue to be published. The clue these results are false positives is that the ‘significant’ results from these small studies generally report effect sizes well outside the range reported for replicated genome-wide significant results from GWAS (Duncan et al. 2019). The lesson is that candidate gene association studies should use genome-wide levels of significance to correct for multiple testing, even when only a small number of variants is genotyped. This would prevent continued publication of false-positive association results. Results should be replicated and, in many cases, this could be easily achieved by looking up the latest relevant GWAS results. Publication of results from small, underpowered candidate gene studies without appropriate statistical analysis and independent replication is no longer justified.

Corrections for multiple testing should also be considered for gene expression studies. The appropriate threshold for microarray or RNA sequence analysis is the subject of some debate. In our studies, we report significance for group comparisons only after multiple testing correction (Fung et al. 2018; Mortlock et al. 2020). Determining an appropriate P-value threshold for statistical significance is critical to differentiate true positives from false positives and false negatives. A key question in expression studies is what number of tests to use for multiple testing correction? In line with experience from genetic studies, we prefer to correct for all genes expressed in the relevant tissue.

There may be a case for choosing a different threshold for some functional studies testing strong hypotheses with good prior evidence. However, we should be wary of relaxed thresholds because many differentially expressed genes from small studies reported to be related to disease pathogenesis do not replicate well in studies with more adequate power. One possible explanation for some false-positive results is the large genetic effects on the expression of many genes (Westra et al. 2013; GTEx Consortium 2017). Unequal genotype frequencies between small samples may explain some differences otherwise attributed to treatment comparisons. Large and better-powered gene expression studies including meta-analyses of results from comparable studies at different centres are likely to provide more reliable estimates of disease-related changes in gene expression, as we have seen for well-powered multicentre GWAS results.

We need to understand the limitations of the methods we use. Too many papers in the reproduction literature report on underpowered studies with misleading conclusions based on poor statistical inference. There remains considerable bias favouring publication of ‘significant’ results over negative results. None of this is helped by the rise in predatory journals with poor editorial standards, but predatory journals are not the only problem. Papers with invalid conclusions based on inadequate study design and poor statistical inference are still published in the top journals in our field. The net result is misinformation or fake news that can easily become dogma, where articles are cited based on their conclusions without scrutiny of the supporting data. Genetic studies on disease risk demonstrate the importance of large studies with adequate power (often requiring collaboration of multiple research groups), combined with rigorous statistical analysis and interpretation. As scientists, reviewers and journal editors, we should reflect on the widespread failure to replicate many published results and insist on higher standards if we want continued support and funding for the vital role research can play in better health care and a productive economy.

The author declares no conflicts of interest.