Choice bias: A quirky byproduct of learning from reward

July 24, 2014

The price of learning from rewarding choices may be just a touch of self-delusion, according to a new study in Neuron.

The research by Brown University brain scientists links a fundamental problem in neuroscience called "credit assignment" – how the brain reinforces learning only in the exact circuits that caused the rewarding choice – to an oft-observed quirk of behavior called "choice bias" – we value the rewards we choose more than equivalent rewards we don't choose. The researchers used computational modeling and behavioral and genetic experiments to discover evidence that choice bias is essentially a byproduct of credit assignment.

"We weren't looking to explain anything about choice bias to start off with," said lead author Jeffrey Cockburn, a graduate student in the research group of senior author Michael Frank, associate professor of cognitive, linguistic and psychological sciences. "This just happened to be the behavioral phenomenon we thought would emerge out of this credit assignment model."

So the next time a friend raves about the movie he chose and is less enthusiastic about the just-as-good one that you chose, you might be able to chalk it up to his basic learning circuitry and a genetic difference that affects it.

Modeled mechanism

The model, developed by Frank, Cockburn and co-author Anne Collins, was based on prior research on the function of the striatum, a part of the brain's basal ganglia (BG) that is principally involved in representing reward values of actions and picking one. "An interaction between three key BG regions moderates that decision making process. When a rewarding choice has been made, the substantia nigra pars compacta (SNc) releases dopamine into the striatum to reinforce connections between cortex and striatum, so that rewarded actions are more likely to be repeated. But how does the SNc reinforce just the circuits that made the right call? The authors proposed a mechanism by which another part of the subtantia nigra, the SNr, detects when actions are worth choosing and then simultaneously amplifies any dopamine signal coming from the SNc."

"The novel part here is that we have proposed a mechanism by which the BG can detect when it has selected an action and should therefore amplify the dopamine reinforcing event specifically at that time," Frank said. "When the SNr decides that striatal valuation signals are strong enough for one action, it releases the brakes not only on downstream structures that allow actions to be executed, but also on the SNc dopamine system, so any unexpected rewards are amplified."

Specifically, dopamine provides reinforcement by enhancing the responsiveness of connections between cells so that a circuit can more easily repeat its rewarding behavior in the future.

But along with that process of reinforcing the action of choosing, the value placed on the resulting reward becomes elevated compared to rewards not experienced this way.

Experimental evidence

That prediction seemed intriguing, but it still had to be tested. The authors, who also included postdoctoral researcher Anne Collins, identified both behavioral and genetic tests that would be telling.

They recruited 80 people at Brown and elsewhere in Providence to play a behavioral game and to donate some saliva for genetic testing.

The game first presented the subjects pictures of arbitrary Japanese characters that would have different probabilities of rewards if chosen ranging from a 20 percent to 80 percent chance of winning a point or losing a point. For some characters, the player could choose a character to discover its resulting reward or penalty, whereas for others, its result was simply given to them. After that learning phase, the subjects were then presented the characters in pairs and instructed to pick the one they thought had the highest chance of winning based on what they had learned.

The researchers built the game so that for every character a player could choose, there was an equally rewarding one had merely been given to them. On average, players showed a clear choice bias in that they were more likely to prefer rewarding characters that they had chosen over equally rewarding characters they had been given.

Notably, they exhibited no choice bias between unrewarding characters suggesting that choice bias emerges only in relation to reward, one of the key predictions of their model. But they wanted to further test whether the impact of reward on choice bias was related to the proposed biological mechanism: that striatal dopaminergic learning is enhanced to chosen rewards.

The genetic tests focused on single-letter differences in a gene called DARPP-32, which governs how well cells in the striatum respond to the reinforcing influence of dopamine.

People with one version of the gene have been shown in previous research to be less able to learn from rewards, while people with other versions were less driven by reward in learning.

"The reason why this gene is interesting is because we know something about the biology of what it does and where it is expressed in the brain," Frank said. "It's predominant in the striatum and specifically affects synaptic plasticity induced by dopamine signaling. It's related to the imbalance to which you learn from really good things, or not so good things.

"The logic was if the mechanism that we think describes this choice bias and credit assignment problem is accurate then that gene should predict the impact of how good something was on this choice bias phenomenon," he said.

Indeed, that's what the data showed. People with the form of the gene that predisposed them to be responsive to big rewards also showed more choice bias from the most strongly rewarded characters. Interestingly, the other people also showed choice bias, but more strongly for those characters that were more mediocre. This pattern was mirrored by the authors' model when it simulated the effects of DARPP-32 on learning imbalances from positive vs. negative outcomes.

For some people, the plums are sweeter if they picked them.

Explore further: Who learns from the carrot, and who from the stick?

Related Stories

Who learns from the carrot, and who from the stick?

November 22, 2013

To flexibly deal with our ever-changing world, we need to learn from both the negative and positive consequences of our behaviour. In other words, from punishment and reward. Hanneke den Ouden from the Donders Institute in ...

When it comes to food, obese women's learning is impaired

July 16, 2014

Obese women were better able to identify cues that predict monetary rewards than those that predict food rewards, according to a study by Yale School of Medicine researchers and their colleagues in the journal Current Biology. ...

Your genes affect your betting behavior

June 16, 2014

Investors and gamblers take note: your betting decisions and strategy are determined, in part, by your genes. University of California, Berkeley, and University of Illinois at Urbana-Champaign (UIUC) researchers have shown ...

Recommended for you

Neurons can learn temporal patterns

May 29, 2017

Individual neurons can learn not only single responses to a particular signal, but also a series of reactions at precisely timed intervals. This is what emerges from a study at Lund University in Sweden.

People match confidence levels to make decisions in groups

May 26, 2017

When trying to make a decision with another person, people tend to match their confidence levels, which can backfire if one person has more expertise than the other, finds a new study led by UCL and University of Oxford researchers.

Optic probes shed light on binge-eating

May 26, 2017

Activating neurons in an area of the brain not previously associated with feeding can produce binge-eating behavior in mice, a new Yale study finds.

Study finds gray matter density increases during adolescence

May 26, 2017

For years, the common narrative in human developmental neuroimaging has been that gray matter in the brain - the tissue found in regions of the brain responsible for muscle control, sensory perception such as seeing and hearing, ...

Game study not playing around with PTSD relief

May 26, 2017

Post-traumatic stress disorder (PTSD) patients wrestling with one of its main symptoms may find long-term relief beyond medication thanks to the work of a Western researcher.

Fathers' brains respond differently to daughters than sons

May 25, 2017

Fathers with toddler daughters are more attentive and responsive to those daughters' needs than fathers with toddler sons are to the needs of those sons, according to brain scans and recordings of the parents' daily interactions ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.