Approach used to conduct meta-analyses may affect outcomes

August 12, 2014

Depending on the analysis strategy used, estimating treatment outcomes in meta­analyses may differ and may result in major alterations in the conclusions derived from the analysis, according to a study in the August 13 issue of JAMA.

Meta-analyses of (RCTs) are generally considered to provide among the best evidence of efficacy of medical interventions. They should be conducted as part of a systematic review, a scientifically rigorous approach that identifies, selects, and appraises all relevant studies. Which to combine in a meta analysis remains a persistent dilemma. Meta-analysis of all trials may produce a precise but biased estimate, according to background information in the article.

Agnes Dechartres, M.D., Ph.D., of the Centre de Recherche Epidemiologie et Statistique, INSERM U1153, Paris, and colleagues compared treatment outcomes estimated by meta-analysis of all trials and several alternative strategies for analysis: single most precise trial (i.e., trial with the narrowest confidence interval), meta-analysis restricted to the 25 percent largest trials, limit meta-analysis (a meta-analysis model adjusted for small-study effect), and meta-analysis restricted to trials at low overall risk of bias. The researchers included 163 meta-analyses published between 2008 and 2010 in high-impact-factor journals and between 2011 and 2013 in the Cochrane Database of Systematic Reviews: 92 (705 RCTs) with subjective outcomes and 71 (535 RCTs) with objective outcomes.

The researchers found that estimates differed depending on the analytic strategy used, with treatment outcomes frequently being larger with meta-analysis of all trials than with the single most precise trial, meta-analysis of the largest trials, and limit meta­analysis. The difference in treatment outcomes between these strategies was substantial in 47 of 92 (51 percent) meta-analyses of subjective outcomes and in 28 of 71 (39 percent) meta-analyses of objective outcomes. The authors did not find any difference in treatment outcomes by overall risk of bias.

"In this study, we compared meta-analysis of all trials with several 'best­evidence' alternative strategies and found that estimated treatment outcomes differed depending on the strategy used. We cannot say which strategy is the best because … we cannot know with 100 percent certainty the truth in any research question. Nevertheless, our results raise important questions about meta-analyses and outline the need to re­think certain principles," the researchers write.

"We recommend that authors of meta-analyses systematically assess the robustness of their results by performing sensitivity analyses. We suggest the comparison of the meta-analysis result to the result for the single most precise trial or meta­analysis of the largest trials and careful interpretation of the meta-analysis result if they disagree."

Jesse A. Berlin, Sc.D., of Johnson & Johnson, Titusville, N.J., and Robert M. Golub, M.D., Deputy Editor, JAMA, write in an accompanying editorial that "findings such as those in the study by Dechartres et al reinforce concerns that journals and readers have about as a study design. Those findings deserve consideration not only in the planning of the studies but in the journal peer review and evaluation. They also reinforce the need for circumspection in study interpretation."

"Meta-analysis has the potential to be the best source of evidence to inform decision making. The underlying methods have become much more sophisticated in the last few decades, but achieving this potential will require continued advances in the underlying science, parallel to the advances that have occurred with other biomedical research design and statistics. Until that occurs, an informed reader must approach these studies, as with all other literature, as imperfect information that requires critical appraisal and assessment of applicability of the findings to individual patients. This is not easy, and it requires skill and intelligence. Whatever clinical evidence looks like, and wherever it is placed on a pyramid, there are no shortcuts to truth."

Explore further: Antidepressant drugs do not improve well-being in children and adolescents

More information: DOI: 10.1001/jama.2014.8166
DOI: 10.1001/jama.2014.8167

Related Stories

Antidepressant drugs do not improve well-being in children and adolescents

July 7, 2014
In an article published in the current issue of Psychotherapy and Psychosomatics the effects of antidepressant drugs on well-being in children and adolescents are analyze.

Mental stress affects ischemia prognosis in patients with CAD

July 2, 2014
(HealthDay)—Mental stress-induced myocardial ischemia (MSIMI) is associated with increased risk of adverse cardiovascular events in patients with coronary artery disease (CAD), according to research published in the July ...

Meta-analysis confirms serum 25(OH)D, mortality link

June 20, 2014
(HealthDay)—Serum 25-hydroxyvitamin (25[OH]D) levels are associated with all-cause and cardiovascular mortality, according to a meta-analysis published online June 17 in BMJ.

Bias pervades the scientific reporting of animal studies

July 17, 2013
A new study published in the open access journal PLOS Biology suggests that the scientific literature could be compromised by substantial bias in the reporting of animal studies, and may be giving a misleading picture of ...

Combined aerobic and resistance training, rather than either method alone, is best for controlling blood sugar

July 2, 2014
A systematic review and meta-analysis of available data published in Diabetologia (the journal of the European Association for the Study of Diabetes) suggests that combined aerobic and resistance training, rather than either ...

Vitamin D supplements have little effect on risk of falls in older people

April 24, 2014
A new meta-analysis, published in The Lancet Diabetes & Endocrinology journal, concludes that there is no evidence to suggest that vitamin D supplements prevent falls, and that ongoing trials to test this theory are unlikely ...

Recommended for you

Scientists find RNA with special role in nerve healing process

August 22, 2017
Scientists may have identified a new opening to intervene in the process of healing peripheral nerve damage with the discovery that an "anti-sense" RNA (AS-RNA) is expressed when nerves are injured. Their experiments in mice ...

Mouse model of human immune system inadequate for stem cell studies

August 22, 2017
A type of mouse widely used to assess how the human immune system responds to transplanted stem cells does not reflect what is likely to occur in patients, according to a study by researchers at the Stanford University School ...

Researchers offer new targets for drugs against fatty liver disease and liver cancer

August 22, 2017
There may no silver bullet for treating liver cancer or fatty liver disease, but knowing the right targets will help scientists develop the most effective treatments. Researchers in Sweden have just identified a number of ...

Make way for hemoglobin

August 18, 2017
Every cell in the body, whether skin or muscle or brain, starts out as a generic cell that acquires its unique characteristics after undergoing a process of specialization. Nowhere is this process more dramatic than it is ...

Bio-inspired materials give boost to regenerative medicine

August 18, 2017
What if one day, we could teach our bodies to self-heal like a lizard's tail, and make severe injury or disease no more threatening than a paper cut?

Are stem cells the link between bacteria and cancer?

August 17, 2017
Gastric carcinoma is one of the most common causes of cancer-related deaths, primarily because most patients present at an advanced stage of the disease. The main cause of this cancer is the bacterium Helicobacter pylori, ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.