Improvements are needed for accuracy in gene-by-environment interaction studies
A new study from McLean Hospital/Harvard Medical School and the University of Colorado concludes that genetic research drawing correlations between specific genes, environmental variables and the combined impact they have on the development of some psychiatric illnesses needs additional scrutiny and replication before being accepted as true.
McLean Hospital investigator Laramie Duncan, PhD, and co-author Matthew Keller, PhD, at the University of Colorado conducted a comprehensive review of the first decade of research looking at how specific genes and environmental variables interact to influence psychiatric disorders including depression, attention deficit hyperactivity disorder (ADHD), alcohol abuse and suicidal behaviors. The study, "A Critical Review of the First 10 Years of Candidate Gene-by-Environment Interaction Research in Psychiatry," is now available online ahead of print publication in the American Journal of Psychiatry.
"Based on our calculations and data from related fields, we estimate that many of the positive findings in this particular area of research may (unintentionally) be incorrect," said Duncan. "What we suggest, to be certain about such correlations, is a focus on the cornerstone of scientific investigation—which is replication. The more we can replicate original findings in follow-up studies, the more we can be sure the results are accurate."
Duncan stressed that her paper is not meant to call for skepticism about the existence of gene-by-environment interactions, or psychiatric research in general, but to shed light on the fact that consistent, replicable results deserve more attention than novel findings and indirect replications.
"Genetic research is like trying to identify a needle in a haystack and statistically, it is predicable that investigators will find and report false positives," said Duncan. "To separate the wheat from the chaff, we need to do follow up studies and replicate the outcomes. That's the only way we can differentiate between accurate findings and the inevitable false positives."
Duncan, who also holds appointments at the Harvard School of Public Health and Harvard Medical School, came upon her research topic while preparing to conduct a study much like the ones she reviewed in her paper. However, in the course of reviewing relevant literature, she noticed trends that prompted her to subject the data to additional statistical tests. When the numbers suggested reason for concern, Duncan's research plans shifted focus.
Duncan and her co-author categorized each of the 103 research studies conducted in the field of gene-by-environment interaction research in psychiatry between 2000 and 2009 as either novel--representing the first reports of particular interactions--or replication studies—efforts to confirm other researchers' results. After analyzing the data, they found that the rate of published significant novel results far outnumbered the rate of replicated results of the same studies. Past reviews have observed that novel findings are more likely to be published but are also more likely to be false positives.
"Upon comparing novel with replication studies, we realized that the many discrepancies are likely a result of publication bias toward positive findings," said Duncan.
According to Duncan and Keller, the tendency to favor publishing significant results over non-significant results is common in research, due both to the desire of publications to showcase groundbreaking findings, and to the decision of authors' not to submit null findings. But, Duncan warns that this bias can be misleading, if one is unaware of it and does not take into account in interpreting the validity of published findings.
"Publication bias is problematic because it produces a distorted representation of findings in an area of study," said Duncan. "Through our research we found that 96 percent of novel studies were significant compared with just 27 percent of replication attempts, suggesting that novel findings appear much stronger than they actually are."
