Choice bias: A quirky byproduct of learning from reward

July 24, 2014

The price of learning from rewarding choices may be just a touch of self-delusion, according to a new study in Neuron.

The research by Brown University brain scientists links a fundamental problem in neuroscience called "credit assignment" – how the brain reinforces learning only in the exact circuits that caused the rewarding choice – to an oft-observed quirk of behavior called "choice bias" – we value the rewards we choose more than equivalent rewards we don't choose. The researchers used computational modeling and behavioral and genetic experiments to discover evidence that choice bias is essentially a byproduct of credit assignment.

"We weren't looking to explain anything about choice bias to start off with," said lead author Jeffrey Cockburn, a graduate student in the research group of senior author Michael Frank, associate professor of cognitive, linguistic and psychological sciences. "This just happened to be the behavioral phenomenon we thought would emerge out of this credit assignment model."

So the next time a friend raves about the movie he chose and is less enthusiastic about the just-as-good one that you chose, you might be able to chalk it up to his basic learning circuitry and a genetic difference that affects it.

Modeled mechanism

The model, developed by Frank, Cockburn and co-author Anne Collins, was based on prior research on the function of the striatum, a part of the brain's basal ganglia (BG) that is principally involved in representing reward values of actions and picking one. "An interaction between three key BG regions moderates that decision making process. When a rewarding choice has been made, the substantia nigra pars compacta (SNc) releases dopamine into the striatum to reinforce connections between cortex and striatum, so that rewarded actions are more likely to be repeated. But how does the SNc reinforce just the circuits that made the right call? The authors proposed a mechanism by which another part of the subtantia nigra, the SNr, detects when actions are worth choosing and then simultaneously amplifies any dopamine signal coming from the SNc."

"The novel part here is that we have proposed a mechanism by which the BG can detect when it has selected an action and should therefore amplify the dopamine reinforcing event specifically at that time," Frank said. "When the SNr decides that striatal valuation signals are strong enough for one action, it releases the brakes not only on downstream structures that allow actions to be executed, but also on the SNc dopamine system, so any unexpected rewards are amplified."

Specifically, dopamine provides reinforcement by enhancing the responsiveness of connections between cells so that a circuit can more easily repeat its rewarding behavior in the future.

But along with that process of reinforcing the action of choosing, the value placed on the resulting reward becomes elevated compared to rewards not experienced this way.

Experimental evidence

That prediction seemed intriguing, but it still had to be tested. The authors, who also included postdoctoral researcher Anne Collins, identified both behavioral and genetic tests that would be telling.

They recruited 80 people at Brown and elsewhere in Providence to play a behavioral game and to donate some saliva for genetic testing.

The game first presented the subjects pictures of arbitrary Japanese characters that would have different probabilities of rewards if chosen ranging from a 20 percent to 80 percent chance of winning a point or losing a point. For some characters, the player could choose a character to discover its resulting reward or penalty, whereas for others, its result was simply given to them. After that learning phase, the subjects were then presented the characters in pairs and instructed to pick the one they thought had the highest chance of winning based on what they had learned.

The researchers built the game so that for every character a player could choose, there was an equally rewarding one had merely been given to them. On average, players showed a clear choice bias in that they were more likely to prefer rewarding characters that they had chosen over equally rewarding characters they had been given.

Notably, they exhibited no choice bias between unrewarding characters suggesting that choice bias emerges only in relation to reward, one of the key predictions of their model. But they wanted to further test whether the impact of reward on choice bias was related to the proposed biological mechanism: that striatal dopaminergic learning is enhanced to chosen rewards.

The genetic tests focused on single-letter differences in a gene called DARPP-32, which governs how well cells in the striatum respond to the reinforcing influence of dopamine.

People with one version of the gene have been shown in previous research to be less able to learn from rewards, while people with other versions were less driven by reward in learning.

"The reason why this gene is interesting is because we know something about the biology of what it does and where it is expressed in the brain," Frank said. "It's predominant in the striatum and specifically affects synaptic plasticity induced by dopamine signaling. It's related to the imbalance to which you learn from really good things, or not so good things.

"The logic was if the mechanism that we think describes this choice bias and credit assignment problem is accurate then that gene should predict the impact of how good something was on this choice bias phenomenon," he said.

Indeed, that's what the data showed. People with the form of the gene that predisposed them to be responsive to big rewards also showed more choice bias from the most strongly rewarded characters. Interestingly, the other people also showed choice bias, but more strongly for those characters that were more mediocre. This pattern was mirrored by the authors' model when it simulated the effects of DARPP-32 on learning imbalances from positive vs. negative outcomes.

For some people, the plums are sweeter if they picked them.

Explore further: Who learns from the carrot, and who from the stick?

Related Stories

Who learns from the carrot, and who from the stick?

November 22, 2013
To flexibly deal with our ever-changing world, we need to learn from both the negative and positive consequences of our behaviour. In other words, from punishment and reward. Hanneke den Ouden from the Donders Institute in ...

Human learning altered by electrical stimulation of dopamine neurons

May 13, 2014
Stimulation of a certain population of neurons within the brain can alter the learning process, according to a team of neuroscientists and neurosurgeons at the University of Pennsylvania. A report in the Journal of Neuroscience ...

When it comes to food, obese women's learning is impaired

July 16, 2014
Obese women were better able to identify cues that predict monetary rewards than those that predict food rewards, according to a study by Yale School of Medicine researchers and their colleagues in the journal Current Biology. ...

Making the right choices in changing circumstances: Cognitive flexibility in the brain

May 27, 2014
Choosing what is best is not always simple. Should one choose a small, certain reward, or take risks and try to get a larger reward? New research by Stan Floresco, from the Brain Research Centre at the University of British ...

Your genes affect your betting behavior

June 16, 2014
Investors and gamblers take note: your betting decisions and strategy are determined, in part, by your genes. University of California, Berkeley, and University of Illinois at Urbana-Champaign (UIUC) researchers have shown ...

Recommended for you

Research revises our knowledge of how the brain learns to fear

October 23, 2017
Our brains wire themselves up during development according to a series of remarkable genetic programs that have evolved over millions of years. But so much of our behavior is the product of things we learn only after we emerge ...

High-speed locomotion neurons found in the brainstem

October 23, 2017
Think of taking a casual stroll on a sunny Sunday afternoon or running at full speed to catch a bus for work on Monday morning as two extremes. Both forms of locomotion entail a perfect interplay between arms and legs, yet ...

Scientists use supercomputer to search for "memory molecules"

October 23, 2017
Until now, searching for genes related to memory capacity has been comparable to seeking out the proverbial "needle in a haystack." Scientists at the University of Basel made use of the CSCS supercomputer Piz Daint to discover ...

Researchers create tool to measure, control protein aggregation

October 22, 2017
A common thread ties seemingly unlinked disorders like Alzheimer's disease and type II diabetes together. This thread is known as protein aggregation and happens when proteins clump together. These complexes are a hallmark ...

Want to control your dreams? Here's how

October 19, 2017
New research at the University of Adelaide has found that a specific combination of techniques will increase people's chances of having lucid dreams, in which the dreamer is aware they're dreaming while it's still happening ...

Researchers find shifting relationship between flexibility, modularity in the brain

October 19, 2017
A new study by Rice University researchers takes a step toward what they see as key to the advance of neuroscience: a better understanding of the relationship between the brain's flexibility and its modularity.

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.