Medical studies with striking results often prove false

October 24, 2012 by Eryn Brown, Los Angeles Times

If a medical study seems too good to be true, it probably is, according to a new analysis.

In a of nearly 230,000 trials compiled from a variety of disciplines, study results that claimed a "very large effect" rarely held up when other research teams tried to replicate them, researchers reported in Wednesday's edition of the .

"The effects largely go away; they become much smaller," said Dr. John Ioannidis, the Stanford researcher who was the report's senior author. "It's likely that most interventions that are effective have modest effects."

Ioannidis and his colleagues came to this conclusion after examining 228,220 trials grouped into more than 85,000 "topics" - collections of studies that paired a single (such as taking a non-steroidal anti- for postoperative pain) with a single outcome (such as experiencing 50 percent relief over six hours). In 16 percent of those topics, at least one study in the group claimed that the intervention made patients at least five times more likely to either benefit or suffer compared with control patients who did not receive the treatment.

In at least 90 percent of those cases, the team found, including data from subsequent trials reduced those odds.

The analysis revealed several reasons to question the significance of the very-large-effect studies, Ioannidis said.

Studies that reported striking results were more likely to be small, with fewer than 100 subjects who experienced fewer than 20 medical events. With such small sample sizes, Ioannidis said, large effects are more likely to be the result of chance.

"Trials need to be of a magnitude that can give useful information," he said.

What's more, the studies that claimed a very large effect tended to measure intermediate effects - for example, whether patients who took a reduced their levels of in their blood - rather than incidence of disease or death itself, outcomes that are more meaningful in assessing medical treatments.

The analysis did not examine individual study characteristics, such as whether the experimental methods were flawed.

The report should remind patients, physicians and policymakers not to give too much credence to small, early studies that show huge treatment effects, Ioannidis said.

One such example: the cancer drug Avastin. Clinical trials suggested the drug might double the time breast cancer patients could live with their disease without getting worse. But follow-up studies found no improvements in progression-free survival, overall survival or patients' quality of life. As a result, the U.S. Food and Drug Administration in 2011 withdrew its approval to use the drug to treat breast cancer, though it is still approved to treat several other types of cancer.

With early glowing reports, Ioannidis said, "one should be cautious and wait for a better trial."

Dr. Rita Redberg, a cardiologist at the University of California, San Francisco who was not involved in the study, said devices and drugs frequently get accelerated approval on the basis of small studies that use intermediate end points.

"Perhaps we don't need to be in such a rush to approve them," she said.

The notion that dramatic results don't hold up under closer scrutiny isn't new. Ioannidis, a well-known critic of the methods used in medical research, has written for years about the ways studies published in peer-reviewed journals fall short. (He's perhaps best known for a 2005 essay in the journal PLoS Medicine titled, "Why Most Published Research Findings Are False.")

But the scope of the JAMA analysis sets it apart from Ioannidis' earlier efforts, said Dr. Gordon Guyatt, a clinical epidemiologist at McMaster University in Hamilton, Canada, who was not involved in the work.

"They looked through a lot of stuff," he said.

Despite widespread recognition that big effects are likely to disappear upon further scrutiny, people still "get excited, and misguidedly so" when presented with home-run results, Guyatt said.

He emphasized that modest effects could benefit patients and were often "very important" on a cumulative basis.

Explore further: Association between biomarkers and disease often overstated, researcher finds

Related Stories

Association between biomarkers and disease often overstated, researcher finds

May 31, 2011
More than two dozen widely cited studies linking genes or other "biomarkers" to specific diseases vastly overstate the association, according to new research from an expert in scientific study design at the Stanford University ...

5 Questions: Ioannidis on the need to test medical 'truths'

January 6, 2012
(Medical Xpress) -- How many established standards of medical care are wrong? Disturbingly, no one knows for sure, but one study suggests that it could be almost half, according to a commentary published in the Jan. 4 issue ...

Stanford researchers suggest ways for physicians to individualize cost-effectiveness of treatments

July 12, 2011
In an era of skyrocketing health-care costs and finite financial resources, health economists are increasingly called upon to determine which medical treatments are the most cost-effective. To do so, they compare the price ...

Most large treatment effects of medical interventions come from small studies

October 23, 2012
In an examination of the characteristics of studies that yield large treatment effects from medical interventions, these studies were more likely to be smaller in size, often with limited evidence, and when additional trials ...

Recommended for you

Comparison of screening recommendations indicates annual mammography

August 21, 2017
When to initiate screening for breast cancer, how often to screen, and how long to screen are questions that continue to spark emotional debates. A new study compares the number of deaths that might be prevented as a result ...

Vitamin C may encourage blood cancer stem cells to die

August 17, 2017
Vitamin C may "tell" faulty stem cells in the bone marrow to mature and die normally, instead of multiplying to cause blood cancers. This is the finding of a study led by researchers from Perlmutter Cancer Center at NYU Langone ...

Outdoor light at night linked with increased breast cancer risk in women

August 17, 2017
Women who live in areas with higher levels of outdoor light at night may be at higher risk for breast cancer than those living in areas with lower levels, according to a large long-term study from Harvard T.H. Chan School ...

Scientists develop novel immunotherapy technology for prostate cancer

August 17, 2017
A study led by scientists at The Wistar Institute describes a novel immunotherapeutic strategy for the treatment of cancer based on the use of synthetic DNA to directly encode protective antibodies against a cancer specific ...

Scientists develop blood test that spots tumor-derived DNA in people with early-stage cancers

August 16, 2017
In a bid to detect cancers early and in a noninvasive way, scientists at the Johns Hopkins Kimmel Cancer Center report they have developed a test that spots tiny amounts of cancer-specific DNA in blood and have used it to ...

Toxic formaldehyde is produced inside our own cells, scientists discover

August 16, 2017
New research has revealed that some of the toxin formaldehyde in our bodies does not come from our environment - it is a by-product of an essential reaction inside our own cells. This could provide new targets for developing ...

2 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

Lurker2358
not rated yet Oct 24, 2012
...rather than incidence of disease or death itself, outcomes that are more meaningful in assessing medical treatments.


Really? Death is pretty universal, and is likely to be so for at least the next several decades, if not quite a bit longer than that. This standard is a bit silly, since many diseases don't actually have a permanent "cure" but only treatments for symptoms or perhaps to slow down the progression.

The analysis did not examine individual study characteristics, such as whether the experimental methods were flawed.


All medical experiments are flawed, because it's impossible to control for everything that could be contaminating the results. Patient age, genetics, lifestyle, diet, exercise, employment, random accidents like unknown chemical or cosmic ray exposure or God only knows what else.

One rides a bike to work, the other rides in a car. The guy on the bike gets lung cancer from breathing unfiltered air, but you don't see a connection in the lab...
riogranderift
not rated yet Oct 25, 2012
Only confirms what we are all taught in training. I hope tax money wasn't spent doing this study. Something bothers me about those who make a living mining data generated by those doing actual research.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.