The role of randomised trials in surgery

December 9, 2016 by Peter Mcculloch, Oxford Science Blog, University of Oxford
The role of randomised trials in surgery. Credit: Shutterstock

In medical science, as in all walks of life, we are impressed by dramatic effects.  If a new treatment seems much better than an old one initially, there is often impatience to get on and use it, and people question why one would want to conduct formal trials.

Doctors who feel this enthusiasm for what they see as a breakthrough often argue that it's not ethical to do a of an exciting new , because the benefits seem so obvious, and randomisation means that half the patients are deprived of them. Of course breakthroughs sometimes turn out to be false dawns, but the idea that something might be so obviously better than what we have now that it doesn't need a randomised trial is pretty widespread in medicine.

We decided to look at this by trying to find all the published randomised trials where the new treatment was reported as being five times better than the previous treatment (or a controlled group). We thought this 'five times better' idea might be a useful rule for medical science. If a hazard ratio of five (i.e. the new treatment is five times better) nearly always predicted correctly that subsequent trials would always report significant benefits, then we could use this as a signpost for the point where no further evidence is needed. Unfortunately, this turned out not to be true.

We studied all of the trials in the Cochrane Collaboration Database (more than 80,000) and found that there were very few instances where there had been both trial with a dramatic effect like this and a subsequent trial. We looked at the ones we found and unfortunately the 'five times better' rule was wrong in over one third of the cases.  In other words, even though an earlier trial showed the new treatment as five times better, a subsequent trial said it was not significantly better at all. We tried to find a rule which worked by increasing the hazard ratio or the significance of the results. We found that we had to increase the hazard ratio to 20 (i.e. the new treatment is 20 times better than the old treatment) before the rule became 100% reliable. Out of the whole Cochrane database there were only four trials that fitted this rule.

So why doesn't this rule work? The main problem is an effect known as 'regression to the mean'.  Most of the trials that show dramatic effects are small trials, and we know that a small trial has a better chance of producing a freak result than a large trial through the effects of pure chance. Smaller randomised trials also tend to be of lower quality than larger ones and therefore open to greater degrees of bias. The implications, particularly for surgery, are quite interesting. It's well known that it's much harder to perform a large randomised trial in surgery than it is when studying a drug. However, our work adds to the literature showing that small randomised are pretty unreliable. Given that they are also very expensive and difficult to do, our results throw into question whether surgeons might be better to do another type of study in situations where they know that they won't be able to do a large enough randomised trial to avoid the effects we are talking about. There will always be exceptions to this , particularly for rare diseases, but our findings could be used to support the idea that in surgery it may be useful sometimes to do a large non-randomised prospective study before committing to the major undertaking of developing a large high quality randomised trial.

Explore further: How to assess the effectiveness of activity trackers for improving health

More information: Myura Nagendran et al. Very large treatment effects in randomised trials as an empirical marker to indicate whether subsequent trials are necessary: meta-epidemiological assessment, BMJ (2016). DOI: 10.1136/bmj.i5432

Related Stories

How to assess the effectiveness of activity trackers for improving health

December 23, 2015
The rise of wearable activity trackers, such as Fitbit, Fuelband, and Jawbone, has generated a lot of public excitement as well as interest from researchers who are enthused about the opportunities these devices may provide ...

Clinical trials: Around half of new treatments perform better than existing treatments

October 16, 2012
On average, new treatments perform better in clinical trials only slightly more often than existing treatments, according to a new systematic review published in The Cochrane Library. The fact that experimental treatments ...

Study suggests prescribing of baclofen for alcohol dependence 'should be reconsidered'

November 30, 2016
The drug baclofen has received high visibility as a possible breakthrough treatment for alcohol dependence. Now a new randomised controlled trial from the University of Amsterdam found no evidence for the usefulness of high-dose ...

Taking additional selenium will not reduce cancer risk

May 11, 2011
Although some people believe that taking selenium can reduce a person's risk of cancer, a Cochrane Systematic Review of randomised controlled clinical trials found no protective effect against non-melanoma skin cancer or ...

Australia's largest trial of ketamine to treat depression begins

August 16, 2016
The largest randomised control trial to evaluate the effectiveness of ketamine as a new treatment for major depression has begun across Australia and New Zealand.

Major clinical trial led to steep declines in use of an ineffective breast cancer treatment

July 7, 2016
Many common medical treatments have never been tested in a clinical trial. A new study published in the July issue of Health Affairs highlights how evidence from trials of untested but widely used treatments can change medical ...

Recommended for you

3-D-printed tracheal splints used in groundbreaking pediatric surgery

September 19, 2018
Children's Healthcare of Atlanta has performed Georgia's first-ever procedure to place 3-D-printed tracheal splints in a pediatric patient. A cross-functional team of Children's surgeons used three custom-made splints, which ...

Muscle relaxants increase risk of respiratory complications

September 18, 2018
Muscle relaxants are a necessary part of anesthesia during certain major operations. However, studies have hinted at respiratory risks connected with these drugs. POPULAR, a major prospective observational European study ...

Gunshot victims require much more blood and are more likely to die than other trauma patients

September 17, 2018
In a new analysis of data submitted to Maryland's state trauma registry from 2005 to 2017, Johns Hopkins Medicine researchers found that gunshot victims are approximately five times more likely to require blood transfusions, ...

Liver allocation system disadvantages children awaiting transplants

September 17, 2018
Children are at a considerable disadvantage when competing with adults for livers from deceased organ donors in the U.S. allocation system, a University of Pittsburgh Graduate School of Public Health-led analysis reveals ...

Taste preferences connected to success of long-term weight loss after bariatric surgery

September 16, 2018
Following Roux-en-Y gastric bypass (RYGB), a type of bariatric surgery, many patients exhibit a reduction in taste preference for sweet and fatty foods, although this effect may only be temporary, according to new research ...

New insights into what drives organ transplant rejection

September 6, 2018
When it comes to transplant rejection, some organs are far trickier than others. Some transplantable organs, such as the liver, are readily accepted by the recipient's immune system, rarely triggering an immune response and ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.