Most large treatment effects of medical interventions come from small studies

October 23, 2012

In an examination of the characteristics of studies that yield large treatment effects from medical interventions, these studies were more likely to be smaller in size, often with limited evidence, and when additional trials were performed, the effect sizes became typically much smaller, according to a study in the October 24/31 issue of JAMA.

"Most effective interventions in confer modest, incremental benefits," according to background information in the article. "Large effects are important to document reliably because in a relative scale they represent potentially the cases in which interventions can have the most impressive effect on and because they are more likely to be adopted rapidly and with less evidence. Consequently, it is important to know whether, when observed, very large effects are reliable and in what sort of experimental outcomes they are commonly observed. … Some large treatment effects may represent entirely spurious observations. It is unknown how often studies with seemingly very large effects are repeated."

Tiago V. Pereira, Ph.D., of the Health Technology Assessment Unit, German Hospital Oswaldo Cruz, Sao Paulo, Brazil, and colleagues conducted a study to evaluate the frequency and features of very large treatment effects of that are first recorded in a clinical trial. For the study, the researchers used data from the Cochrane Database of Systematic Reviews (CDSR) and assessed the types of treatments and outcomes in trials with very large effects, examined how often large-effect trials were followed up by other trials on the same topic, and how these effects compared against the effects of the respective meta-analyses.

Among 3,545 available reviews, 3,082 contributed usable information on 85,002 forest plots (a graphical display designed to illustrate the relative strength of treatment effects in multiple studies). Overall, 8,239 forest plots (9.7 percent) had a nominally statistically significant very large effect in the first published trial, group A; 5,158 (6.1 percent) had a nominally statistically significant very large effect found only after the first published trial, group B; and 71,605 (84.2 percent) had no trials with significant very large effects, group C. The researchers found that nominally significant very large effects arose mostly from small trials with few events. For the index trials, the median [midpoint] number of events was only 18 in group A and 15 in the group B. The median number of events in the group C index trials was 14.

The authors also observed that 90 percent and 98 percent of the very large effects observed in first and subsequently published trials, respectively, became smaller in meta-analyses that included other trials; the median odds ratio decreased from approximately 12 to 4 for first trials, and from 10 to 2.5 for subsequent trials.

Topics with very large effects were less likely than other topics to address mortality. Across the whole CDSR, there was only 1 intervention with large beneficial effects on mortality and no major concerns about the quality of the evidence (for a trial on extracorporeal oxygenation for severe respiratory failure in newborns).

"… this empirical evaluation suggests that very large effect estimates are encountered commonly in single trials. Conversely, genuine very large effects with extensive support from substantial evidence appear to be rare in medicine and large benefits for mortality are almost entirely nonexistent. As additional evidence accumulates, caution may still be needed, especially if there is repetitive testing of accumulating trials. Patients, clinicians, investigators, regulators, and the industry should consider this in evaluating very large treatment effects when the evidence is still early and weak," the researchers write.

Explore further: Statins are unlikely to prevent blood clots, analysis finds

More information: JAMA. 2012;308[16]:1676-1684

Related Stories

Statins are unlikely to prevent blood clots, analysis finds

September 18, 2012
Despite previous studies suggesting the contrary, statins (cholesterol-lowering drugs) may not prevent blood clots (venous thrombo-embolism) in adults, according to a large analysis by international researchers published ...

Music reduces anxiety in cancer patients

August 10, 2011
(Medical Xpress) -- Cancer patients may benefit from sessions with trained music therapists or from listening to music. A new Cochrane systematic review shows using music can reduce anxiety in cancer patients, and may also ...

Recommended for you

Cell cycle-blocking drugs can shrink tumors by enlisting immune system in attack on cancer

August 16, 2017
In the brief time that drugs known as CDK4/6 inhibitors have been approved for the treatment of metastatic breast cancer, doctors have made a startling observation: in certain patients, the drugs—designed to halt cancer ...

Toxic formaldehyde is produced inside our own cells, scientists discover

August 16, 2017
New research has revealed that some of the toxin formaldehyde in our bodies does not come from our environment - it is a by-product of an essential reaction inside our own cells. This could provide new targets for developing ...

Researchers find 'switch' that turns on immune cells' tumor-killing ability

August 16, 2017
Molecular biologists led by Leonid Pobezinsky and his wife and research collaborator Elena Pobezinskaya at the University of Massachusetts Amherst have published results that for the first time show how a microRNA molecule ...

Popular immunotherapy target turns out to have a surprising buddy

August 16, 2017
The majority of current cancer immunotherapies focus on PD-L1. This well studied protein turns out to be controlled by a partner, CMTM6, a previously unexplored molecule that is now suddenly also a potential therapeutic target. ...

A metabolic treatment for pancreatic cancer?

August 15, 2017
Pancreatic cancer is now the third leading cause of cancer mortality. Its incidence is increasing in parallel with the population increase in obesity, and its five-year survival rate still hovers at just 8 to 9 percent. Research ...

Skewing the aim of targeted cancer therapies

August 15, 2017
Headlines, of late, have touted the successes of targeted gene-based cancer therapies, such as immunotherapies, but, unfortunately, also their failures.

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.