Potential neurological treatments often advance to clinical trials on shaky evidence, study says

July 16, 2013

Clinical trials of drug treatments for neurological diseases such as Alzheimer's and Parkinson's often fail because the animal studies that preceded them were poorly designed or biased in their interpretation, according to a new study from an international team of researchers. More stringent requirements are needed to assess the significance of animal studies before testing the treatments in human patients, the researchers say.

The team - led by John Ioannidis, MD, DSc, a professor of medicine at the Stanford University School of Medicine and an expert in - assessed the results of more than 4,000 animal studies in 160 meta-analyses of potential treatments for neurological disorders from Alzheimer's disease, Parkinson's disease, stroke, spinal-cord injury and a form of multiple sclerosis. (A meta-analysis is a study that compiles and assesses information and conclusions from many independent experiments of a treatment, or intervention, for a particular condition.).

They determined that only eight of the 160 studies of potential treatments yielded the statistically significant, unbiased data necessary to support advancing the treatment to clinical trials. In contrast, 108 of the treatments were deemed at least somewhat effective at the time they were published.

Ioannidis and his at the University of Edinburgh in Scotland and the University of Ioannina School of Medicine in Greece say that animal studies of potential interventions can be made more efficient and reliable by increasing average sample size, being aware of statistical bias, publishing negative results and making all the results of all experiments on the effectiveness of a particular treatment - regardless of their outcome - freely accessible to scientists.

"Some researchers have postulated that animals may not be good models for human diseases," said Ioannidis. "I don't agree. I think animal studies can be useful and perfectly fine. The problem is more likely to be related to the selective availability of information about the studies conducted on animals." Although the researchers focused here on neurological disorders, they believe it is likely that similar bias exists in animal studies of other types of disorders.

Ioannidis, who directs the Stanford Prevention Research Center, is the senior author of the research, which will be published online in PLoS Biology on July 16. Lecturer Konstantinos Tsilidis, PhD, and postgraduate fellow Orestis Panagiotou, MD, of the University of Ioannina share lead authorship of the study. Panagiotou is currently a researcher at the National Cancer Institute's Division of Cancer Epidemiology and Genetics.

Ioannidis is known for his efforts to strengthen the way that research is planned, carried out and reported. He was called "one of the world's foremost experts on the credibility of medical research" in a profile published in The Atlantic magazine in 2010. He outlined some of the problems he observed in a 2005 essay in PLoS-Medicine titled, "Why most published research findings are false." The essay is one of the most-downloaded articles in the history of the Public Library of Science, according to the journal's media relations office.

For the new study, Ioannidis and his colleagues evaluated results in a database of the thousands of animal studies compiled over the years through the CAMARADES initiative (Collaborative Approach to Meta-Analysis and Review of Animal Data in Experimental Studies), led by professor Malcolm MacLeod, PhD, from the University of Edinburgh, who is also a co-author of the study.

The team compared the number of experiments in the meta-analyses that would have been expected to yield positive results (based on their predicted statistical power) with the actual number of experiments with published positive results. The difference was striking: 919 expected versus the 1,719 that were published, implying that either negative results were not published, or that the results of the experiments were interpreted too optimistically.

"We saw that it was very common for these interventions to have published evidence that they would work," said Ioannidis. "It was extremely common to have results that suggest they would be effective in humans."

Furthermore, nearly half (46 percent) of the 160 showed evidence of small-study effects - a term used to describe the fact that a small study using fewer numbers of animals is more likely to find the intervention more effective than a larger study with many animals.

Ioannidis speculated that a reluctance to publish negative findings (that is, those that conclude that a particular intervention did not work any better than the control treatment) and a perhaps unconscious desire on the part of researchers to find a promising treatment has colored the field of neurological research. Obscuring access to studies that conclude a particular treatment is ineffective, while also publishing positive results that are likely to be statistically flawed, tilts the perception toward the potential effectiveness of an intervention and encourages unwarranted human clinical trials.

"There are no standard rules that guide a decision to move from animal studies into human clinical trials," said Ioannidis, who also holds C.F. Rehnborg Professorship at Stanford. "Sometimes interventions are tested in humans with very little evidence that they may be effective. Of the 160 analyses we studied, only eight had what we would call strong evidence of potential effectiveness with no hint of bias in the preliminary animal studies. And of these eight, only two have given positive results in humans."

Ioannidis believes the development of consortiums of groups of researchers studying a particular intervention, coupled with the free sharing of all data about its effectiveness, or lack thereof, is a good first step in reducing bias in animal studies.

"Under the current conditions, only a tiny proportion of interventions that have published some promising results in animals have shown to be at all effective in humans. For example, while dozens of treatments on ischemic or hemorrhagic stroke seem to work in the animal literature, almost none of them have worked in humans," said Ioannidis. "It is hard to believe we could not improve upon that translation record. If we raise the bar for moving into human trials, centralize researchers' efforts and make all results available, it will be much easier for researchers to know whether they have a potential winner, and it would increase the efficiency of human enormously."

Explore further: Medical studies with striking results often prove false

More information: Tsilidis KK, Panagiotou OA, Sena ES, Aretouli E, Evangelou E, et al. (2013) Evaluation of Excess Significance Bias in Animal Studies of Neurological Diseases. PLoS Biol 11(7): e1001609. doi:10.1371/journal.pbio.1001609

Related Stories

Medical studies with striking results often prove false

October 24, 2012
If a medical study seems too good to be true, it probably is, according to a new analysis.

Association between biomarkers and disease often overstated, researcher finds

May 31, 2011
More than two dozen widely cited studies linking genes or other "biomarkers" to specific diseases vastly overstate the association, according to new research from an expert in scientific study design at the Stanford University ...

Stanford researchers suggest ways for physicians to individualize cost-effectiveness of treatments

July 12, 2011
In an era of skyrocketing health-care costs and finite financial resources, health economists are increasingly called upon to determine which medical treatments are the most cost-effective. To do so, they compare the price ...

5 Questions: Ioannidis on the need to test medical 'truths'

January 6, 2012
(Medical Xpress) -- How many established standards of medical care are wrong? Disturbingly, no one knows for sure, but one study suggests that it could be almost half, according to a commentary published in the Jan. 4 issue ...

Pets battling cancer can join clinical trials too

June 14, 2013
(HealthDay)—If you hear that a friend's beloved family member has joined a clinical trial for cancer treatment, don't assume the patient is human.

Recommended for you

Make way for hemoglobin

August 18, 2017
Every cell in the body, whether skin or muscle or brain, starts out as a generic cell that acquires its unique characteristics after undergoing a process of specialization. Nowhere is this process more dramatic than it is ...

Bio-inspired materials give boost to regenerative medicine

August 18, 2017
What if one day, we could teach our bodies to self-heal like a lizard's tail, and make severe injury or disease no more threatening than a paper cut?

Are stem cells the link between bacteria and cancer?

August 17, 2017
Gastric carcinoma is one of the most common causes of cancer-related deaths, primarily because most patients present at an advanced stage of the disease. The main cause of this cancer is the bacterium Helicobacter pylori, ...

Two-step process leads to cell immortalization and cancer

August 17, 2017
A mutation that helps make cells immortal is critical to the development of a tumor, but new research at the University of California, Berkeley suggests that becoming immortal is a more complicated process than originally ...

Female mouse embryos actively remove male reproductive systems

August 17, 2017
A protein called COUP-TFII determines whether a mouse embryo develops a male reproductive tract, according to researchers at the National Institutes of Health and their colleagues at Baylor College of Medicine, Houston. The ...

New Pathology Atlas maps genes in cancer to accelerate progress in personalized medicine

August 17, 2017
A new Pathology Atlas is launched today with an analysis of all human genes in all major cancers showing the consequence of their corresponding protein levels for overall patient survival. The difference in expression patterns ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.