Brain imaging results skewed by biased study samples

October 12, 2017
Credit: CC0 Public Domain

Any scientist who studies groups of people knows that the characteristics of the "sample"—the group of people selected for the study—can profoundly impact the study's findings. To produce the most accurate findings, a study group ought to be as similar as possible to the people in the larger population you want to say something about.

A new UC San Francisco-led study shows that failure to follow this basic principle of population science—a common complaint about research in the cognitive sciences—can profoundly skew the results of brain imaging studies, leading to errors that may be throwing off neuroscientists' understanding of normal .

"Much of what we know about how the brain develops comes from samples that don't look like the broader U.S. at all," said Kaja LeWinn, ScD, an epidemiologist and assistant professor of psychiatry at UCSF, member of the UCSF Weill Institute for Neurosciences, and lead author of the new study. "We would never try to understand the burden of other health conditions, like cardiovascular disease, in a sample with much higher socioeconomic status than the U.S. population as a whole, for instance."

In recent years, social scientists have drawn attention to the over-representation of so-called W.E.I.R.D (White, Educated, Industrial, Rich, Developed) individuals in cognitive science experiments, but LeWinn says her team's new study—published online October 12, 2017 in Nature Communications—is the first to look directly at exactly how these sampling practices impact the research results of neuroimaging studies.

The research team—which included Margaret A. Sheridan, PhD, of the University of North Carolina at Chapel Hill; Katherine M. Keyes, PhD, and Ava Hamilton of Columbia University; and Katie A. McLaughlin, PhD, of the University of Washington—found that applying the sampling approaches frequently used in brain imaging studies to a large dataset of pediatric MRI images significantly distorted findings about how different regions of the brain develop with age.

"These findings give us pause because they raise questions about existing knowledge of brain development in children, which is based almost entirely on non-representative samples," LeWinn said. "These foundational studies inform our understanding not only of brain development, but also of mental health disorders like depression and autism. It will be critical for the field to evaluate whether these findings replicate in larger samples that better resemble the US population in all its diversity."

The new findings are also likely to have broad implications for understanding why it is often difficult to reproduce research results in human neuroscience, LeWinn added. "The field has had a hard time replicating findings in neuroimaging, but so far, explanations for that have been focused on the way that are processed and analyzed, not on sample composition. But if we conducted the exact same study on two groups of people with completely different underlying characteristics, there is no reason to expect that we would find the same answer."

The Developing Brain Changes With Age—and With Study Samples

The authors made use of a large, publicly available dataset of brain scans from 1162 children aged three to 18 years from the Pediatric Imaging, Neurocognition and Genetics Study (PING). As is typical of developmental neuroimaging studies, the demographics of the PING sample were not very similar to the wider U.S. population. For example, household education and income levels were considerably higher in the children in this dataset than in the broader U.S. Typically, investigators would use PING data without considering these demographic differences, LeWinn said.

In the new study LeWinn and her colleagues created a "weighted" version of this dataset that better reflected the distribution of demographic characteristics such as sex, race/ethnicity, parental education and income in the US population, and then asked whether correctly weighting the sample changed what the brain images revealed about how the brain develops during childhood.

The researchers found that weighting the data in this fashion significantly altered what the brain images showed about the timeline of brain development during childhood and adolescence. In most cases, the weighted data showed brain regions reaching their peak size much earlier than the unweighted data would have suggested, suggesting a faster pace of development in the weighted sample. For example, the unweighted data showed overall brain surface area peaking at 12.1 years , while the correctly weighted data revealed that this peak more likely occurs at 9.7 years of age—a 2.4 year difference.

The authors also found that weighting the PING dataset impacted the apparent order in which different brain regions develop. In the unweighted sample, the frontal, occipital, and temporal lobes of the brain all seem to develop at about the same time, which is contrary to a broader understanding in neuroscience that the brain develops from back to front, starting with sensory and motor regions and only in late adolescence finishing the development of regions in the front of the brain involved in more abstract forms of cognition. The weighted imaging data matched this back-to-front model much better: it showed the parietal and occipital lobes in the back of the head developing earliest, followed by the temporal lobe beneath the ears, and the frontal lobe last.

"The differences we found are pretty dramatic," said LeWinn. "The weighted data showed maturation peaking in some as much as four years earlier than the unweighted data would have suggested. We've shown that simply by weighting a dataset to better approximate a representative U.S. sample, we can end up with very different ideas about how the brain develops. Not only do we see big differences in age at peak surface area and volume, but we also found that the weighted data gives us a pattern of development that is more consistent with what we'd expect, with sensorimotor areas developing first, followed by association cortices."

Enter Population Neuroscience

The new research adds to a growing debate in neuroscience about the use of small "samples of convenience" rather than investing in larger and more sweeping samples that carefully represent well-defined target populations of interest within the U.S.

"Most cognitive neuroscience studies are so small that we can't apply the weighting methods used in our study to approximate a U.S. sample, which we were able to do because of the large size of the PING dataset," LeWinn said. "But at a minimum, researchers doing these studies need to be very transparent about the demographic characteristics of their sample, and consider to whom it's reasonable to generalize their findings. That way when the community sees inconsistent findings across studies of the same research question, we can consider differences in sample composition as one potential explanation."

LeWinn added that it's not reasonable to insist that all neuroimaging studies of healthy populations be perfectly representative of the U.S: "That would be impractical and financially prohibitive, and we certainly don't want to suggest that. But certainly for large, multisite studies, like the new NIH-funded ABCD [Adolescent Brain Cognitive Development] study, which represents a major investment of taxpayer dollars, representative sampling should be a primary goal."

Explore further: Our brains do change from early to mid-adulthood

More information: Kaja Z. LeWinn et al, Sample composition alters associations between age and brain structure, Nature Communications (2017). DOI: 10.1038/s41467-017-00908-7

Related Stories

Our brains do change from early to mid-adulthood

August 21, 2017
Scientists in China have found that significant microstructural changes occur in the brain from early to mid-adulthood, allowing them to accurately estimate an individual's age from their brain structure. The findings are ...

Childhood maltreatment may change brain's response to threat

September 11, 2017
Neural activity associated with defensive responses in humans shifts between two brain regions depending on the proximity of a threat, suggests neuroimaging data from two independent samples of adults in the Netherlands published ...

New findings on brain functional connectivity may lend insights into mental disorders

August 31, 2017
Ongoing advances in understanding the functional connections within the brain are producing exciting insights into how the brain circuits function together to support human behavior—and may lead to new discoveries in the ...

Abnormalities shown to first appear in brain networks involved in sensory processing

August 29, 2017
The origins of autism remain mysterious. What areas of the brain are involved, and when do the first signs appear? New findings published in Biological Psychiatry brings us closer to understanding the pathology of autism, ...

Study finds gray matter density increases during adolescence

May 26, 2017
For years, the common narrative in human developmental neuroimaging has been that gray matter in the brain - the tissue found in regions of the brain responsible for muscle control, sensory perception such as seeing and hearing, ...

Cerebellum plays a major role in schizophrenia

June 22, 2017
In a new study, Norwegian researchers have documented that the cerebellum is among the most affected brain regions in schizophrenia. Compared to healthy individuals, cerebellar volume was smaller in patients with schizophrenia. ...

Recommended for you

'Selfish brain' wins out when competing with muscle power, study finds

October 20, 2017
Human brains are expensive - metabolically speaking. It takes lot of energy to run our sophisticated grey matter, and that comes at an evolutionary cost.

Researchers find shifting relationship between flexibility, modularity in the brain

October 19, 2017
A new study by Rice University researchers takes a step toward what they see as key to the advance of neuroscience: a better understanding of the relationship between the brain's flexibility and its modularity.

Brain training can improve our understanding of speech in noisy places

October 19, 2017
For many people with hearing challenges, trying to follow a conversation in a crowded restaurant or other noisy venue is a major struggle, even with hearing aids. Now researchers reporting in Current Biology on October 19th ...

Investigating the most common genetic contributor to Parkinson's disease

October 19, 2017
LRRK2 gene mutations are the most common genetic cause of Parkinson's disease (PD), but the normal physiological role of this gene in the brain remains unclear. In a paper published in Neuron, Brigham and Women's Hospital ...

Want to control your dreams? Here's how

October 19, 2017
New research at the University of Adelaide has found that a specific combination of techniques will increase people's chances of having lucid dreams, in which the dreamer is aware they're dreaming while it's still happening ...

Brain takes seconds to switch modes during tasks

October 19, 2017
The brain rapidly switches between operational modes in response to tasks and what is replayed can predict how well a task will be completed, according to a new UCL study in rats.

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.