Bias pervades the scientific reporting of animal studies

A new study published in the open access journal PLOS Biology suggests that the scientific literature could be compromised by substantial bias in the reporting of animal studies, and may be giving a misleading picture of the chances that potential treatments could work in humans.

Testing a new therapeutic intervention (such as a drug or surgical procedure) on human subjects is expensive, risky and ethically complex, so the vast majority are first tested on animals. Unfortunately, cost and constrain the size of animal studies, giving them limited statistical power, and as a result the scientific literature contains many studies that are either uncertain in their outcomes or contradictory. A way around this limitation has been to conduct a "meta-analysis": scientists collect data from a large number of published studies on the same intervention, combine them using sophisticated statistical methods, and then end up with a much more solid basis on which to decide whether to proceed with human clinical trials.

In the new study, Konstantinos Tsilidis, John Ioannidis and colleagues at Stanford University examined 160 previously published meta-analyses of animal studies looking at potential treatments for a range of serious human neurological disorders (, stroke, Parkinson's disease, Alzheimer's disease and injury). These meta-analyses covered 1000 original published animal studies comparing more than 4000 sets of . The authors' "meta-analysis of meta-analyses" used the most precise study in each meta-analysis as an estimate of the true effect size of a particular treatment. It then asked whether the expected number of studies had statistically significant conclusions. Alarmingly, the authors found that more than twice as many studies as expected appeared to reach statistical significance.

The authors suggest that rather than reflecting wilful fraud on the part of the scientists who conduct the original studies, this "excess significance bias" comes from two main sources. One is that scientists conducting an animal study tend to choose the method of data analysis that appears to give them the "better" result. The second arises because scientists usually want to publish in higher profile journals; such journals tend to strongly prefer studies with positive, rather than negative, results. Many studies with negative results are not even submitted for publication or, if submitted, either cannot get published or are published belatedly in low-visibility journals, reducing their chances of inclusion in a meta-analysis.

It is likely that the types of bias reported in the new PLOS Biology paper have been responsible for the inappropriate promotion of treatments from animal studies into human clinical trials. It also seems unlikely that this phenomenon is confined to studies of neurological disorders; rather this is probably a general feature of the reporting of animal studies.

The authors suggest several remedies for the bias that they have observed. First, animal studies should adhere to strict guidelines (such as the ARRIVE guidelines) for study design and analysis. Second, animal studies (like human clinical trials) should be pre-registered so that publication of the outcome, however negative, is ensured. Third, availability of methodological details and raw data would make it easier for other scientists to verify published studies.

More information: Tsilidis KK, Panagiotou OA, Sena ES, Aretouli E, Evangelou E, et al. (2013) Evaluation of Excess Significance Bias in Animal Studies of Neurological Diseases. PLoS Biol 11(7): e1001609. doi:10.1371/journal.pbio.1001609

add to favorites email to friend print save as pdf

Related Stories

Meta-analysis helps psychologists build knowledge

Jan 09, 2012

(Medical Xpress) -- When scientists want to know the answer to a question that’s been studied a great deal, they conduct something called a meta-analysis, pooling data from multiple studies to arrive at one combined ...

Recommended for you

What are the chances that your dad isn't your dad?

Apr 16, 2014

How confident are you that the man you call dad is really your biological father? If you believe some of the most commonly-quoted figures, you could be forgiven for not being very confident at all. But how ...

New technology that is revealing the science of chewing

Apr 15, 2014

CSIRO's 3D mastication modelling, demonstrated for the first time in Melbourne today, is starting to provide researchers with new understanding of how to reduce salt, sugar and fat in food products, as well ...

After skin cancer, removable model replaces real ear

Apr 11, 2014

(HealthDay)—During his 10-year struggle with basal cell carcinoma, Henry Fiorentini emerged minus his right ear, and minus the hearing that goes with it. The good news: Today, the 56-year-old IT programmer ...

Italy scraps ban on donor-assisted reproduction

Apr 09, 2014

Italy's Constitutional Court on Wednesday struck down a Catholic Church-backed ban against assisted reproduction with sperm or egg donors that has forced thousands of sterile couples to seek help abroad.

User comments