Poor methodology reporting makes radiation oncology studies impossible to replicate

Nearly 80 percent of radiation oncology studies funded by the National Institutes of Health involve investigating the effects that radiation has on tumor cells and healthy tissue in pre-clinical settings, such as experiments done in cell cultures or mice. A majority of these radiation biology studies, however, have serious flaws in how their irradiation methodology is described, which makes them very difficult to replicate, according to a new finding from the University of Maryland School of Medicine (UMSOM).

Important details in the irradiation protocol and the experimental setup are routinely not included in most of these journal articles, which could lead to dose variations or other errors when other researchers try to repeat the experiments in their own laboratories.

"Glaring omissions or errors in the methodology sections of radiation biology journal articles make the studies very difficult to reproduce, interpret, and compare with other research," said study principal investigator Yannick Poirier, Ph.D., Assistant Professor in the Department of Radiation Oncology at UMSOM.

In the new study published this month in the International Journal of Radiation Oncology Biology and Physics, Dr. Poirier and his colleagues from UMSOM and the University of Washington, Seattle, reviewed 1,758 peer-reviewed studies from 469 journals that were used in pre-clinical studies to evaluate the effects of radiation therapy to treat malignant tumors and other conditions. Specifically, they examined the journal article's "methods" section that described the radiation protocol used.

The researchers found that the source of radiation used in the study (such as low vs high energy X-rays or gamma rays) was unclear or ambiguous in nearly 14 percent of the studies they reviewed. They also found that only 1 percent of studies listed the protocol the researchers used to calibrate the machines, and only 16 percent named the equipment used to measure the absorbed radiation dose.

Several publications were found to contain outright errors where a quantity of radiation dose was misreported. In a few cases, researchers found descriptions of "unachievable" experiments including irradiation produced by a linear accelerator at energy levels that would have been impossible for the named device to produce.

The authors graded the level of physics reporting necessary to successfully reproduce the experiment on a scale of 1(worst) to 10 (best) and found that only 3 percent of the journal articles would receive a score of 8 or above. "Disturbingly, articles published in higher impact journals—and, consequently, those that are higher cited—scored the poorest. This means that these studies with poor-quality physics are being propagated and are amplifying the reproducibility problem," said study co-author Amit Sawant, Ph.D., Associate Professor and Chief of the Division of Physics in the Department of Radiation Oncology at UMSOM.

The UMSOM researchers stipulate that their findings are limited to the physics and irradiation aspects of these radiation biology studies. The quality of the science, hypotheses, or non-physics aspects of the experimental design were not reviewed by the UMSOM researchers.

"This inability to reproduce scientific findings could profoundly impact the translation of preclinical research results into clinical practice," said E. Albert Reece, MD, Ph.D., MBA, Executive Vice President for Medical Affairs, UM Baltimore, and the John Z. and Akiko K. Bowers Distinguished Professor and Dean, University of Maryland School of Medicine. "The UMSOM researchers highlight a crucial issue that should be addressed by the radiation oncology research community."

The failure to report or reproduce radiation absorbed dose in radiation biology journal articles is likely due to a lack of consultation between the radiation biologists performing the study and radiation physicists who have the expertise to design, document and validate the radiation delivery protocol, according to the UMSOM researchers.

"We are raising awareness of this issue because the problem can be fixed by involving more properly trained physicists in these studies," said Dr. Poirier. "That is because radiation delivery follows well-understood radiation physics principles that lend themselves to high accuracy, precision, and reproducibility."

Efforts are already underway to address the problem: NIH's National Cancer Institute and National Institute on Allergy and Infectious Diseases recently formed a program to standardize and monitor the radiation dosimetry delivery among the entirety of their preclinical research program funded by the radiation countermeasures program. The American Association of Physicists also established a task force last year to create guidelines for accurate dosimetry in radiobiology experiments.