Massive study reports challenges in reproducing published psychology findings

August 27, 2015, University of Virginia
Credit: Charles Rondeau/public domain

A study that sought to replicate 100 findings published in three prominent psychology journals has found that, across multiple criteria, independent researchers could replicate less than half of the original findings. In some cases this may call into question the validity of some scientific findings, but it may also point to the difficulty of conducting effective replications and achieving reproducible results.

The results of this review study, conducted by more than 270 researchers on five continents, are published in the Aug. 28 issue of the journal Science. Twenty-two students and faculty from the University of Virginia were among the co-authors.

"For years there has been concern about the reproducibility of scientific findings, but little direct, systematic evidence. This project is the first of its kind and adds substantial evidence that the concerns are real and addressable," said Brian Nosek, a U.Va. psychology professor and coordinator of the study.

Nosek is the co-founder and executive director of the Center for Open Science, which coordinated the Reproducibility Project: Psychology. The project has produced the most comprehensive, open investigation ever about the rate and predictors of reproducibility in a field of science.

Reproducibility means that the results recur when the same data are analyzed again, or when new data are collected using the same methods.

"With this project we established an initial estimate of the rate of reproducibility in psychology, and identified some evidence of possible influences on reproducibility," said Anup Gampa, a Reproducibility Project team member and Ph.D. candidate at U.Va. "This sets the stage for new research to examine how to improve reproducibility."

Science is unique from other ways of gaining knowledge, Gampa said, because it relies on reproducibility to gain confidence in ideas and evidence.

"Scientific evidence does not rely on trusting the authority of the person who made the discovery," said Reproducibility Project team member Angela Attwood, a at the University of Bristol. "Rather, credibility accumulates through independent replication and elaboration of the ideas and evidence."

However, Elizabeth Gilbert, a Reproducibility Project team member and Ph.D. candidate at U.Va., noted that a failure to reproduce does not necessarily mean the original report was incorrect.

"A replication team must have a complete understanding of the methodology used for the original research, and shifts in the context or conditions of the research could be unrecognized but important for observing the result," she said.

Nosek pointed out that a problem for psychology, as well as in other disciplines, is that incentives for scientists are not consistently aligned with reproducibility.

"Scientists aim to contribute reliable knowledge, but also need to produce results that help them keep their job as a researcher," he said. "To thrive in science, researchers need to earn publications, and some kinds of results are easier to publish than others, particularly ones that are novel and show unexpected or exciting new directions."As a consequence, according to Nosek and his co-authors, many scientists pursue innovative research in the interest of their careers, even at the cost of reproducibility of the findings. The authors say that research

with new, surprising findings is more likely to be published than research examining when, why or how existing findings can be reproduced.

Overall, the Reproducibility Project team successfully replicated fewer than half of the original findings. Investigators suggested this could be due to three basic reasons:

  1. Though most replication teams worked with the original authors to use the same materials and methods, small differences in when, where or how the replication was carried out might have influenced the results.
  2. The replication might have failed, by chance, to detect the original result.
  3. The original result might have been a false positive.

"The findings demonstrate that reproducing original results may be more difficult than is presently assumed, and interventions may be needed to improve reproducibility," said Johanna Cohoon, a project coordinator with the Charlottesville-based Center for Open Science.

In keeping with the goals of openness and reproducibility, each replication project team posted its methods and results on a public website.

Many organizations, funders, journals and publishers are working to improve . The journal Psychological Science, one of the publications included in this study, last year implemented practices to make study materials and data readily and openly available to other researchers.

"Efforts include increasing transparency of original research materials, code and data so that other teams can more accurately assess, replicate and extend the original research, and pre-registration of research designs to increase the robustness of the inferences drawn from the statistical analyses applied to research results," said Denny Borsboom, a project team member from the University of Amsterdam who was involved in the creation of the Transparency and Openness Promotion Guidelines, recently published in Science.

Explore further: Psychologists collaborate to reproduce experimental results

More information: Estimating the reproducibility of psychological science, www.sciencemag.org/lookup/doi/ … 1126/science.aac4716

Related Stories

Psychologists collaborate to reproduce experimental results

November 28, 2013
(Medical Xpress)—Psychological studies have a bad reputation for delivering results that researchers never reproduce. Repeatedly, psychologists have been unable to replicate the effects of classic studies. To test the reproducibility ...

Recommended for you

Imagining a positive outcome biases subsequent memories

April 24, 2018
Imagining that a future event will go well may lead you to remember it more positively after it's over, according to findings from research published in Psychological Science, a journal of the Association for Psychological ...

Depressed, inactive and out of work—study reveals lives of lonely young adults

April 24, 2018
New research from King's College London shows that lonely young adults are more likely to experience mental health problems and more likely to be out of work than their peers. The study, published today in Psychological Medicine, ...

Sense of control and meaning helps protect women from anxiety

April 24, 2018
People who feel in control of their lives and who find purpose and meaning in life are less likely to have anxiety disorders even when going through the toughest times, according to a study led by the University of Cambridge.

Researchers identify brain mechanism linking PTSD and opioid addiction

April 23, 2018
Researchers at Western University have shown that the recall of traumatic memories enhances the rewarding effects of morphine, shedding light on the neurobiological link between post-traumatic stress disorder (PTSD) and opioid ...

Scientific guidelines for using cannabis to treat stress, anxiety and depression

April 19, 2018
In a first-of-a-kind study, Washington State University scientists examined how peoples' self-reported levels of stress, anxiety and depression were affected by smoking different strains and quantities of cannabis at home.

Neuroscientists use magnetic stimulation to amplify PTSD therapy

April 19, 2018
Researchers from The University of Texas at Dallas have found that a standard therapy for post-traumatic stress disorder (PTSD) is more effective when paired with transcranial magnetic stimulation of the brain.

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.