The role of randomised trials in surgery

December 9, 2016 by Peter Mcculloch, Oxford Science Blog, University of Oxford
The role of randomised trials in surgery. Credit: Shutterstock

In medical science, as in all walks of life, we are impressed by dramatic effects.  If a new treatment seems much better than an old one initially, there is often impatience to get on and use it, and people question why one would want to conduct formal trials.

Doctors who feel this enthusiasm for what they see as a breakthrough often argue that it's not ethical to do a of an exciting new , because the benefits seem so obvious, and randomisation means that half the patients are deprived of them. Of course breakthroughs sometimes turn out to be false dawns, but the idea that something might be so obviously better than what we have now that it doesn't need a randomised trial is pretty widespread in medicine.

We decided to look at this by trying to find all the published randomised trials where the new treatment was reported as being five times better than the previous treatment (or a controlled group). We thought this 'five times better' idea might be a useful rule for medical science. If a hazard ratio of five (i.e. the new treatment is five times better) nearly always predicted correctly that subsequent trials would always report significant benefits, then we could use this as a signpost for the point where no further evidence is needed. Unfortunately, this turned out not to be true.

We studied all of the trials in the Cochrane Collaboration Database (more than 80,000) and found that there were very few instances where there had been both trial with a dramatic effect like this and a subsequent trial. We looked at the ones we found and unfortunately the 'five times better' rule was wrong in over one third of the cases.  In other words, even though an earlier trial showed the new treatment as five times better, a subsequent trial said it was not significantly better at all. We tried to find a rule which worked by increasing the hazard ratio or the significance of the results. We found that we had to increase the hazard ratio to 20 (i.e. the new treatment is 20 times better than the old treatment) before the rule became 100% reliable. Out of the whole Cochrane database there were only four trials that fitted this rule.

So why doesn't this rule work? The main problem is an effect known as 'regression to the mean'.  Most of the trials that show dramatic effects are small trials, and we know that a small trial has a better chance of producing a freak result than a large trial through the effects of pure chance. Smaller randomised trials also tend to be of lower quality than larger ones and therefore open to greater degrees of bias. The implications, particularly for surgery, are quite interesting. It's well known that it's much harder to perform a large randomised trial in surgery than it is when studying a drug. However, our work adds to the literature showing that small randomised are pretty unreliable. Given that they are also very expensive and difficult to do, our results throw into question whether surgeons might be better to do another type of study in situations where they know that they won't be able to do a large enough randomised trial to avoid the effects we are talking about. There will always be exceptions to this , particularly for rare diseases, but our findings could be used to support the idea that in surgery it may be useful sometimes to do a large non-randomised prospective study before committing to the major undertaking of developing a large high quality randomised trial.

Explore further: How to assess the effectiveness of activity trackers for improving health

More information: Myura Nagendran et al. Very large treatment effects in randomised trials as an empirical marker to indicate whether subsequent trials are necessary: meta-epidemiological assessment, BMJ (2016). DOI: 10.1136/bmj.i5432

Related Stories

How to assess the effectiveness of activity trackers for improving health

December 23, 2015
The rise of wearable activity trackers, such as Fitbit, Fuelband, and Jawbone, has generated a lot of public excitement as well as interest from researchers who are enthused about the opportunities these devices may provide ...

Clinical trials: Around half of new treatments perform better than existing treatments

October 16, 2012
On average, new treatments perform better in clinical trials only slightly more often than existing treatments, according to a new systematic review published in The Cochrane Library. The fact that experimental treatments ...

Study suggests prescribing of baclofen for alcohol dependence 'should be reconsidered'

November 30, 2016
The drug baclofen has received high visibility as a possible breakthrough treatment for alcohol dependence. Now a new randomised controlled trial from the University of Amsterdam found no evidence for the usefulness of high-dose ...

Taking additional selenium will not reduce cancer risk

May 11, 2011
Although some people believe that taking selenium can reduce a person's risk of cancer, a Cochrane Systematic Review of randomised controlled clinical trials found no protective effect against non-melanoma skin cancer or ...

Australia's largest trial of ketamine to treat depression begins

August 16, 2016
The largest randomised control trial to evaluate the effectiveness of ketamine as a new treatment for major depression has begun across Australia and New Zealand.

Major clinical trial led to steep declines in use of an ineffective breast cancer treatment

July 7, 2016
Many common medical treatments have never been tested in a clinical trial. A new study published in the July issue of Health Affairs highlights how evidence from trials of untested but widely used treatments can change medical ...

Recommended for you

Blood stored longer may be less safe for patients with massive blood loss and shock

March 10, 2018
Human blood from donors can be stored for use up to 42 days, and it is a mainstay therapy in transfusion medicine. However, recent studies looking back at patient records have shown that transfusion with older, stored blood ...

After knee replacement, play on

March 8, 2018
(HealthDay)—Knee replacement patients can continue to enjoy sports—such as skiing, tennis and dancing—without worrying that high-impact activities might compromise their new joint, a small, new study finds.

Engineered cartilage template to heal broken bones

March 8, 2018
A team of UConn Health researchers has designed a novel, hybrid hydrogel system to help address some of the challenges in repairing bone in the event of injury. The UConn Health team, led by associate professor of orthopedic ...

Treating hypothyroidism to stop a stubborn surgical complication

March 7, 2018
Researchers at the University of Michigan have discovered a link between low thyroid hormone levels and wound healing complications.

Neurocognitive impairment linked to worse outcomes after total joint replacement

March 6, 2018
People with undiagnosed neurocognitive deficits are undergoing hip and knee replacements at high rates and are more likely to have poorer short-term outcomes after surgery, according to new research led by orthopedic surgeons ...

Minimally invasive surgeries underused in older patients, new study finds

March 5, 2018
A study of more than 200,000 Medicare patients who had common surgical procedures shows that, compared to the general population, they underwent far fewer minimally invasive operations, whose benefits include lower rates ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.