Researchers call for tougher standards for studies on obesity policies

May 1, 2018, Johns Hopkins University

When a new park is built, a tax is instituted on fast food or a ban put in place against soft drinks in a school, public health researchers must often rely on "after the fact" observational studies to evaluate the impact of such efforts on rates of obesity in a particular population and try to clearly identify and measure the factors that worked or didn't.

Such so-called natural experiments are distinct from randomized controlled trials, the gold standard experiment in which groups are randomly selected to be part of an intervention or not.

Now, Johns Hopkins researchers report that a review of nearly 300 studies purporting to evaluate the impact of antiobesity programs, policies and environmental changes suggests that while natural experiments have a lot to offer, the field needs to agree on some stricter standards for assessing the effects of broad public health policies.

A summary of the findings is published April 30 in the Annals of Internal Medicine.

"Our findings about the strengths and weaknesses of natural experiments are in part a call to action for better standards," says Wendy Bennett, M.D., M.P.H., associate professor of medicine at the Johns Hopkins University School of Medicine. "We conclude that researchers need to use stronger study designs and more standardized reporting methods so that we can better identify policies and programs that work in our schools and communities," she adds.

Bennett says an estimated 1.9 billion people worldwide are overweight or obese, a condition contributing to epidemics of heart disease, diabetes, high blood pressure and cancer. The financial costs of are high, but so also are public health interventions, she notes.

In an effort to get a better view of what works and doesn't work, Bennett and her colleagues identified 294 studies, including 156 (53 percent) natural experiments, reporting the effects of antiobesity programs and policies. Of the 294 studies, 118 (40 percent) were classified as standard experimental studies, and 20 (7 percent) had unclear study designs.

For each study, two reviewers independently extracted data on the population characteristics, data sources, measures, analytical methods and risks of bias in study design and outcomes.

"It's important that we know what researchers are doing, what types of data they're using and what they're measuring when they're asking these questions about obesity interventions," says Bennett. "Then we can ask how we can improve on that so we have valid and trustworthy results."

Most studies, Bennett's team found, used large data sources such as national or statewide surveys—including the California Healthy Kids Survey and the National Health and Nutrition Examination Survey in adults. However, even when multiple databases were used, they were rarely similar or linked together.

Outcomes studied in the natural experiments included populationwide changes in body weight, body mass index (BMI), physical activity, caloric intake, fruit and vegetable intake, sugar-sweetened beverage intake and fast-food frequency. In natural experiments, the most common study design—used in more than a third of the studies—was to compare a group exposed to an intervention, such as a ban on sweetened school drinks, with a group not exposed to it. Another 31 percent of natural experiments compared the same population before and after a policy, program or environment change, most commonly only at two time points.

Of the classical experimental studies, 55 percent were rated as having a strong study design, and 61 percent adequately addressed confounding variables—the idea that some characteristics about a population or policy were not considered in the design of the study and could have a hidden effect on the results. Overall, Bennett's team found, natural experiments often (64 percent) fell short in these areas.

"We often didn't see comparisons between exposed and unexposed groups to see if they were really appropriate to compare," says Bennett. "If one community or school had a policy change and the other didn't, were those communities or schools really comparable to begin with? We can't always tell from the study report."

Only 28 percent of natural experiments, for instance, were rated "strong" when it came to participant selection bias, and only 44 percent were considered "strong" when it came to addressing confounding variables. Overall, nearly 80 percent of natural experiments were rated as having a high risk of bias due to one or more study design factors.

"When researchers publish results, they need to do so in a way that makes clear what they measured and what the risk of bias is," says Bennett. "We need to establish better standards around reporting for natural experiment studies."

Bennett adds that public health researchers working in this area also need to improve data collection methods—self-reporting phone or paper surveys are notoriously weak, for example—and enhance the links between existing data sources; train future researchers in more depth when it comes to methods for natural experiments; and design a central clearinghouse for studies.

"Educators have the What Works Clearinghouse where you can go in and see what teaching methods and curricula are most effective," says Bennett. "For obesity control and prevention, we need something similar to catalog and critique the studies that are being done. We want to know which are the best studies showing effective policies and programs to reduce and prevent obesity in our communities."

Explore further: Review: need to strengthen natural experiments in obesity

More information: Wendy L. Bennett et al. Methods for Evaluating Natural Experiments in Obesity, Annals of Internal Medicine (2018). DOI: 10.7326/M18-0309

Related Stories

Review: need to strengthen natural experiments in obesity

May 1, 2018
(HealthDay)—Based on findings from a systematic review of the literature, recommendations have been developed for improving the evidence for natural experiments in obesity; the review and position paper were published online ...

From soda bans to bike lanes: Which 'natural experiments' really reduce obesity?

March 20, 2015
Banning sodas from school vending machines, building walking paths and playgrounds, adding supermarkets to food deserts and requiring nutritional labels on restaurant menus: Such changes to the environments where people live ...

Mandatory nutrition policies may impact sugar consumption

March 28, 2018
Mandatory nutrition policies could be a valuable tool in helping high school students to lower their sugar intake, a University of Waterloo study has found.

Physician support key to successful weight loss, study shows

August 21, 2015
A review of survey data from more than 300 obese people who participated in a federally funded weight loss clinical trial found that although the overall weight loss rates were modest, those who rated their primary care doctor's ...

Warning labels can help reduce soda consumption and obesity, new study suggests

December 15, 2017
Labels that warn people about the risks of drinking soda and other sugar-sweetened beverages can lower obesity and overweight prevalence, suggests a new Johns Hopkins Bloomberg School of Public Health study.

Reporting of dietary intake methods in obesity trials poor

August 23, 2012
(HealthDay)—More care needs to be taken in reporting dietary intake methods in childhood and adolescent obesity intervention trials in order to be able to better evaluate and replicate study methods, according to the results ...

Recommended for you

Obesity linked to increased risk of early-onset colorectal cancer

October 12, 2018
Women who are overweight or obese have up to twice the risk of developing colorectal cancer before age 50 as women who have what is considered a normal body mass index (BMI), according to new research led by Washington University ...

The metabolome: A way to measure obesity and health beyond BMI

October 11, 2018
The link between obesity and health problems may seem apparent. People who are obese are at higher risk of type 2 diabetes, liver disease, cancer, and heart disease. But increasingly, researchers are learning that the connection ...

Being overweight or obese in your 20s will take years off your life, according to a new report

October 10, 2018
Young adults classified as obese in Australia can expect to lose up to 10 years in life expectancy, according to a major new study.New modelling from The George Institute for Global Health and the University of Sydney also ...

Asthma may contribute to childhood obesity epidemic

October 9, 2018
Toddlers with asthma are more likely to become obese children, according to an international study led by USC scientists.

'Genes are not destiny' when it comes to weight

October 9, 2018
A healthy home environment could help offset children's genetic susceptibilities to obesity, according to new research led by UCL.

What did americans eat today? A third would say fast food

October 3, 2018
(HealthDay)—Americans' love affair with fast food continues, with 1 in every 3 adults chowing down on the fare on any given day.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.