The role of randomised trials in surgery

December 9, 2016 by Peter Mcculloch, Oxford Science Blog, University of Oxford
The role of randomised trials in surgery. Credit: Shutterstock

In medical science, as in all walks of life, we are impressed by dramatic effects.  If a new treatment seems much better than an old one initially, there is often impatience to get on and use it, and people question why one would want to conduct formal trials.

Doctors who feel this enthusiasm for what they see as a breakthrough often argue that it's not ethical to do a of an exciting new , because the benefits seem so obvious, and randomisation means that half the patients are deprived of them. Of course breakthroughs sometimes turn out to be false dawns, but the idea that something might be so obviously better than what we have now that it doesn't need a randomised trial is pretty widespread in medicine.

We decided to look at this by trying to find all the published randomised trials where the new treatment was reported as being five times better than the previous treatment (or a controlled group). We thought this 'five times better' idea might be a useful rule for medical science. If a hazard ratio of five (i.e. the new treatment is five times better) nearly always predicted correctly that subsequent trials would always report significant benefits, then we could use this as a signpost for the point where no further evidence is needed. Unfortunately, this turned out not to be true.

We studied all of the trials in the Cochrane Collaboration Database (more than 80,000) and found that there were very few instances where there had been both trial with a dramatic effect like this and a subsequent trial. We looked at the ones we found and unfortunately the 'five times better' rule was wrong in over one third of the cases.  In other words, even though an earlier trial showed the new treatment as five times better, a subsequent trial said it was not significantly better at all. We tried to find a rule which worked by increasing the hazard ratio or the significance of the results. We found that we had to increase the hazard ratio to 20 (i.e. the new treatment is 20 times better than the old treatment) before the rule became 100% reliable. Out of the whole Cochrane database there were only four trials that fitted this rule.

So why doesn't this rule work? The main problem is an effect known as 'regression to the mean'.  Most of the trials that show dramatic effects are small trials, and we know that a small trial has a better chance of producing a freak result than a large trial through the effects of pure chance. Smaller randomised trials also tend to be of lower quality than larger ones and therefore open to greater degrees of bias. The implications, particularly for surgery, are quite interesting. It's well known that it's much harder to perform a large randomised trial in surgery than it is when studying a drug. However, our work adds to the literature showing that small randomised are pretty unreliable. Given that they are also very expensive and difficult to do, our results throw into question whether surgeons might be better to do another type of study in situations where they know that they won't be able to do a large enough randomised trial to avoid the effects we are talking about. There will always be exceptions to this , particularly for rare diseases, but our findings could be used to support the idea that in surgery it may be useful sometimes to do a large non-randomised prospective study before committing to the major undertaking of developing a large high quality randomised trial.

Explore further: How to assess the effectiveness of activity trackers for improving health

More information: Myura Nagendran et al. Very large treatment effects in randomised trials as an empirical marker to indicate whether subsequent trials are necessary: meta-epidemiological assessment, BMJ (2016). DOI: 10.1136/bmj.i5432

Related Stories

How to assess the effectiveness of activity trackers for improving health

December 23, 2015
The rise of wearable activity trackers, such as Fitbit, Fuelband, and Jawbone, has generated a lot of public excitement as well as interest from researchers who are enthused about the opportunities these devices may provide ...

Clinical trials: Around half of new treatments perform better than existing treatments

October 16, 2012
On average, new treatments perform better in clinical trials only slightly more often than existing treatments, according to a new systematic review published in The Cochrane Library. The fact that experimental treatments ...

Study suggests prescribing of baclofen for alcohol dependence 'should be reconsidered'

November 30, 2016
The drug baclofen has received high visibility as a possible breakthrough treatment for alcohol dependence. Now a new randomised controlled trial from the University of Amsterdam found no evidence for the usefulness of high-dose ...

Taking additional selenium will not reduce cancer risk

May 11, 2011
Although some people believe that taking selenium can reduce a person's risk of cancer, a Cochrane Systematic Review of randomised controlled clinical trials found no protective effect against non-melanoma skin cancer or ...

Australia's largest trial of ketamine to treat depression begins

August 16, 2016
The largest randomised control trial to evaluate the effectiveness of ketamine as a new treatment for major depression has begun across Australia and New Zealand.

Major clinical trial led to steep declines in use of an ineffective breast cancer treatment

July 7, 2016
Many common medical treatments have never been tested in a clinical trial. A new study published in the July issue of Health Affairs highlights how evidence from trials of untested but widely used treatments can change medical ...

Recommended for you

Surgical adhesions can be treated, prevented in mice

November 28, 2018
A cellular culprit—as well as a possible treatment—for a common, sometimes life-threating post-surgical complication has been identified by researchers at the Stanford University School of Medicine.

Cost and weight-loss potential matter most to bariatric surgery patients

November 28, 2018
A JAMA Surgery study found that patients are likely to base their weight loss surgery choice on expected out-of-pocket costs, and how much weight they can lose—not risk of complications or recovery time.

Treating spinal pain with replacement discs made of 'engineered living tissue' moves closer to reality

November 21, 2018
For the first time, bioengineered spinal discs were successfully implanted and provided long-term function in the largest animal model ever evaluated for tissue-engineered disc replacement. A new Penn Medicine study published ...

Screening for colorectal cancer spares male patients from intense treatments

November 21, 2018
While screening for colorectal cancer does not reduce mortality, it does reduce the need for chemotherapy and emergency surgeries among male patients, according to a recent Finnish study.

Rapid response inpatient education boosts use of needed blood-thinning drugs

November 16, 2018
A new study designed to reach hospitalized patients at risk shows that a "real-time" educational conversation, video or leaflet can lower the missed dose rates of drugs that can prevent potentially lethal blood clots in their ...

Race plays role in regaining weight after gastric bypass surgery

November 15, 2018
African Americans and Hispanic Americans who have undergone Roux-en-Y gastric bypass (RYGB) are at greater risk to regain weight as compared to Caucasians. To date, no study has addressed the effect of race on weight regain ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.