Medical studies with striking results often prove false
If a medical study seems too good to be true, it probably is, according to a new analysis.
In a statistical analysis of nearly 230,000 trials compiled from a variety of disciplines, study results that claimed a "very large effect" rarely held up when other research teams tried to replicate them, researchers reported in Wednesday's edition of the Journal of the American Medical Association.
"The effects largely go away; they become much smaller," said Dr. John Ioannidis, the Stanford researcher who was the report's senior author. "It's likely that most interventions that are effective have modest effects."
Ioannidis and his colleagues came to this conclusion after examining 228,220 trials grouped into more than 85,000 "topics" - collections of studies that paired a single medical intervention (such as taking a non-steroidal anti-inflammatory drug for postoperative pain) with a single outcome (such as experiencing 50 percent relief over six hours). In 16 percent of those topics, at least one study in the group claimed that the intervention made patients at least five times more likely to either benefit or suffer compared with control patients who did not receive the treatment.
In at least 90 percent of those cases, the team found, including data from subsequent trials reduced those odds.
The analysis revealed several reasons to question the significance of the very-large-effect studies, Ioannidis said.
Studies that reported striking results were more likely to be small, with fewer than 100 subjects who experienced fewer than 20 medical events. With such small sample sizes, Ioannidis said, large effects are more likely to be the result of chance.
"Trials need to be of a magnitude that can give useful information," he said.
What's more, the studies that claimed a very large effect tended to measure intermediate effects - for example, whether patients who took a statin drug reduced their levels of bad cholesterol in their blood - rather than incidence of disease or death itself, outcomes that are more meaningful in assessing medical treatments.
The analysis did not examine individual study characteristics, such as whether the experimental methods were flawed.
The report should remind patients, physicians and policymakers not to give too much credence to small, early studies that show huge treatment effects, Ioannidis said.
One such example: the cancer drug Avastin. Clinical trials suggested the drug might double the time breast cancer patients could live with their disease without getting worse. But follow-up studies found no improvements in progression-free survival, overall survival or patients' quality of life. As a result, the U.S. Food and Drug Administration in 2011 withdrew its approval to use the drug to treat breast cancer, though it is still approved to treat several other types of cancer.
With early glowing reports, Ioannidis said, "one should be cautious and wait for a better trial."
Dr. Rita Redberg, a cardiologist at the University of California, San Francisco who was not involved in the study, said devices and drugs frequently get accelerated approval on the basis of small studies that use intermediate end points.
"Perhaps we don't need to be in such a rush to approve them," she said.
The notion that dramatic results don't hold up under closer scrutiny isn't new. Ioannidis, a well-known critic of the methods used in medical research, has written for years about the ways studies published in peer-reviewed journals fall short. (He's perhaps best known for a 2005 essay in the journal PLoS Medicine titled, "Why Most Published Research Findings Are False.")
But the scope of the JAMA analysis sets it apart from Ioannidis' earlier efforts, said Dr. Gordon Guyatt, a clinical epidemiologist at McMaster University in Hamilton, Canada, who was not involved in the work.
"They looked through a lot of stuff," he said.
Despite widespread recognition that big effects are likely to disappear upon further scrutiny, people still "get excited, and misguidedly so" when presented with home-run results, Guyatt said.
He emphasized that modest effects could benefit patients and were often "very important" on a cumulative basis.
(c)2012 Los Angeles Times
Distributed by MCT Information Services
- New guidelines issued for reporting of genetic risk research Mar 28, 2011 | not rated yet | 0
- Association between biomarkers and disease often overstated, researcher finds May 31, 2011 | not rated yet | 0
- 5 Questions: Ioannidis on the need to test medical 'truths' Jan 06, 2012 | not rated yet | 0
- Stanford researchers suggest ways for physicians to individualize cost-effectiveness of treatments Jul 12, 2011 | not rated yet | 0
- Most large treatment effects of medical interventions come from small studies Oct 23, 2012 | not rated yet | 0
- Motion perception revisited: High Phi effect challenges established motion perception assumptions Apr 23, 2013 | 3 / 5 (2) | 2
- Anything you can do I can do better: Neuromolecular foundations of the superiority illusion (Update) Apr 02, 2013 | 4.5 / 5 (11) | 5
- The visual system as economist: Neural resource allocation in visual adaptation Mar 30, 2013 | 5 / 5 (2) | 9
- Separate lives: Neuronal and organismal lifespans decoupled Mar 27, 2013 | 4.9 / 5 (8) | 0
- Sizing things up: The evolutionary neurobiology of scale invariance Feb 28, 2013 | 4.8 / 5 (10) | 14
Why is zone 1 in liver more prone to ischemic injury?
May 23, 2013 Hi, Is it because around central vein, there is only deoxygenated blood from the vein where as in the periphery there is hepatic artery. Also why...
How can there be villous adenoma in colon, if there are no villi there
May 22, 2013 As title suggest. Thanks :smile:
How can there be a term called "intestinal metaplasia" of stomach
May 21, 2013 Hello everyone, Ok Stomach's normal epithelium is simple columnar, now in intestinal type of adenocarcinoma of stomach it undergoes "intestinal...
Pressure-volume curve: Elastic Recoil Pressure don't make sense
May 18, 2013 From pressure-volume curve of the lung and chest wall (attached photo), I don't understand why would the elastic recoil pressure of the lung is...
If you became brain-dead, would you want them to pull the plug?
May 17, 2013 I'd want the rest of me to stay alive. Sure it's a lousy way to live but it beats being all-the-way dead. Maybe if I make it 20 years they'll...
MRI bill question
May 15, 2013 Dear PFers, The hospital gave us a $12k bill for one MRI (head with contrast). The people I talked to at the hospital tell me that they do not...
- More from Physics Forums - Medical Sciences
More news stories
In recent years, microRNAs (miRNAs) and other non-coding RNAs are small molecules that help control the expression of specific proteins. In recent years they have emerged as disease biomarkers. miRNA profiles have been used ...
Cancer May 24, 2013 | not rated yet | 0
Cancer cells spread and grow by avoiding detection and destruction by the immune system. Stimulation of the immune system can help to eliminate cancer cells; however, there are many factors that cause the immune system to ...
Cancer May 24, 2013 | 5 / 5 (1) | 0
Researchers from London's Kingston University have begun a two-year study which could help prolong the lives of people with colorectal tumours.
Cancer May 24, 2013 | 5 / 5 (1) | 0
Transformative research from Western University has identified new hormones in the body which may suppress breast cancer and stimulate the regression of breast tumors.
Cancer May 24, 2013 | 5 / 5 (2) | 0
(Medical Xpress)—Curtin University researchers have found evidence that targeting specific cells in the body can reverse the effects of cancer on the immune system.
Cancer May 24, 2013 | 5 / 5 (4) | 0
Coenzyme Q10 decreases all cause mortality by half, according to the results of a multicentre randomised double blind trial presented today at Heart Failure 2013 congress. It is the first drug to improve heart failure mortality ...
19 hours ago | 5 / 5 (5) | 5
Heart failure accelerates the aging process and brings on early andropausal syndrome (AS), according to research presented today at the Heart Failure Congress 2013. AS, also referred to as male 'menopause', was four times ...
19 hours ago | 5 / 5 (1) | 1
(HealthDay)—Animals make great companions for senior citizens, but elderly people who always drive with a pet in the car are far more likely to crash than those who never drive with a pet, researchers have ...
11 hours ago | not rated yet | 1
(Medical Xpress)—A research team, led by Jeremy Barr, a biology post-doctoral fellow, unveils a new immune system that protects humans and animals from infection.
May 20, 2013 | 4.8 / 5 (31) | 9 |
Mortality and length of stay are highest in heart failure patients admitted in January, on Friday, and overnight, according to research presented today at the Heart Failure Congress 2013. The analysis of nearly 1 million ...
19 hours ago | not rated yet | 0
(AP)—Department of Justice lawyers have again asked a federal appeals court in New York to delay lifting age restrictions and prescription requirements on an emergency contraceptive popularly known as the morning-after ...
19 hours ago | not rated yet | 0