For a less biased study, try randomization

The researchers found that certain kinds of studies — those whose recruiters were more likely to know in which comparison group the next participant would land — had larger estimates of effects than those that whose “blinding” was considered adequate. Yet the finding does not mean such studies are inaccurate.

The findings could give researchers more insight into the pros and cons of the strategies they embrace to figure out whether one drug or treatment is better than another. While it might seem obvious that a randomized study is the most reliable option, scientists do not universally believe it is, said review lead author Jan Odgaard-Jensen.

“Although there is wide agreement about the value of randomized trials, some people still argue that randomized trials may not be more reliable than non-randomized studies,” said Odgaard-Jensen, a statistician with the Norwegian Knowledge Center for the Health Service.

Even if researchers do acknowledge the superiority of randomized studies, many turn to a non-randomized approach that is cheaper and less complicated. For example, they could try to allocate patients in a study in a way that is not entirely random, such as assigning them to treatments based on ZIP code or gender, Odgaard-Jensen said.

So how much of a difference does randomization make in the results of a study? The review authors sought to find out by updating the results of a previous review.

The researchers deemed 18 papers — all reviews or meta-analyses — to fit their criteria to include in the review, which appears in the latest issue of The Cochrane Library.

The journal is a publication of The Cochrane Collaboration, an international organization that evaluates medical research. Systematic reviews draw evidence-based conclusions about medical practice after considering both the content and quality of existing medical trials on a topic.

The review authors found that randomized and non-randomized studies did not consistently yield a certain kind of result. Sometimes the estimates of effect — a measurement of, say, whether a medication works — were larger in randomized studies. However, they were larger in some non-randomized studies too.

The review authors write that, in general, it is not possible to figure out how biases in the kinds of studies could throw off the results. In other words, one cannot assume that a randomized controlled trial that yields a certain estimates of effect is more or less accurate than one that does not randomly assign participants, which yields a different estimate.

Dr. Gordon Guyatt, a professor of medicine at McMaster University, in Ontario, Canada, said the review findings confirm that we do not live in a “simpler and more easy to handle world” in which “biases would always go in the same direction.”

“It turns out that the world is not a simple place,” he said. “Sometimes the bias goes in one direction and sometimes in another.”

However, the unpredictability of randomization in a randomized study — when it’s impossible to know beforehand whether a patient will end up in one group or another — adds to the value of that specific approach, review lead author Odgaard-Jensen said. It is “the best protection against the unpredictability of the extent and direction of bias in clinical trials that are not properly randomized.”

Still, randomized trials are not guaranteed routes to the truth regarding a medical question, said Dr. Arthur Hartz, director of health service research at Huntsman Cancer Institute in Salt Lake City. “I don’t think any single study is conclusive. You often get randomized controlled trials that disagree with each other.”

Randomized studies are often a nuisance too, he said. “They take incredible amounts of time and money, and they’re boring, and by the time you get the results, medical technology has moved on.”

One alternative is an observational study, in which researchers do not control which subjects get a particular medication or treatment, he said. (The new review did not examine the value of observational studies.)

“We have some indications of where observational studies are giving bad results, but we don’t understand exactly why they give bad results and how it could be fixed,” Hartz said. “That’s where the emphasis should be.”

For his part, Guyatt said randomization is one of several safeguards against bias in studies, such as preventing various players from knowing who is getting which treatment and following up with as many subjects as possible to see what happened to them. “If all these bias-reducing strategies are in place, randomized trials will always get you closer to the truth than studies that don’t have all the safeguards,” he said.

Ultimately, said Tufts University professor of medicine Dr. Joseph Lau, the new review “has value to those developing tools to critically appraise studies in systematic reviews. It also has value to those proposing guidelines on improving the conduct and reporting of primary studies.”

More information: Odgaard-Jensen J, et al. Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Systematic Reviews 2011, Issue 4.

Provided by Health Behavior News Service