Researchers test new Risk of Bias assessment tool for non-randomized intervention studies

The Cochrane Risk of Bias Tool for Non-Randomized Studies of Interventions shows potential value in systematic reviews in an assessment published this week in PLOS Medicine. The study, conducted by researchers at Dalla Lana School of Public Health at the University of Toronto, Ontario, Canada demonstrated that exclusion of studies with higher Risk of Bias (RoB) assessed by a new instrument developed by the international Cochrane Collaboration changed estimates of clinically pertinent outcomes in two published systematic reviews of cardiovascular risks associated with use of two popular classes of drugs.

Inclusion of studies at high RoB may lead to erroneous conclusions about the benefits and harms of medical interventions. Researchers undertaking and meta-analysis (SRMA) of usually measure RoB of the component studies using an established tool, but there is no widely-accepted measure of RoB for non-randomized studies. As a test of performance, the Toronto-based researchers applied the new Cochrane RoB instrument, called ACROBAT-NRSI, to 37 papers included in two widely cited systematic reviews of the use of thiazolidinediones ("glitazones" used to treat diabetes) and cyclooxygenase-2 (COX-2) inhibitors (used to treat inflammation), and risk of major cardiovascular events ( and heart failure) and death. Assessment took roughly 2.5 hours per study, and inter-rater agreement was moderate to substantial (overall Kappa 0.45 to 1.0). Of the 37 studies originally included in the meta-analyses, only eight had low RoB scored by ACROBAT-NRSI. Confining meta-analysis to studies with low RoB shifted the pooled odds ratio for myocardial infarction for rosiglitazone versus pioglitazone from 1.14 [95% CI 1.07-1.24] to 1.06 [95% CI 0.99-1.13]). Estimates of pooled relative risks (RR) of cardiovascular events with COX-2 inhibitors compared with no nonsteroidal anti-inflammatory drug changed little when analyses were confined to studies with low or moderate RoB, with the exception of an increased RR associated with ibuprofen (from 1.07 (95% CI 0.97-1.18) to 1.14 (95% CI 1.03-1.26)).

Because the study was conducted by a specific two-person team and considered only two systematic reviews, the findings may not be generalizable and further testing of the new Cochrane instrument is needed. Nonetheless, the findings indicate that at least two published meta-analyses included studies with high RoB. The authors state, "[e]xclusion of studies deemed to have unacceptably high RoB may impact the findings of pooled estimates of intervention effects, altering both the statistical and clinical significance of the results."

More information: Bilandzic A, Fitzpatrick T, Rosella L, Henry D (2016) Risk of Bias in Systematic Reviews of Non-Randomized Studies of Adverse Cardiovascular Effects of Thiazolidinediones and Cyclooxygenase-2 Inhibitors: Application of a New Cochrane Risk of Bias Tool. PLoS Med 13(4): e1001987. DOI: 10.1371/journal.pmed.1001987

Journal information: PLoS Medicine
Citation: Researchers test new Risk of Bias assessment tool for non-randomized intervention studies (2016, April 5) retrieved 20 April 2024 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

Review explores harms linked to antidepressant treatment


Feedback to editors