Brain mechanism teaches mice to avoid bullies: Findings may offer insight into human social disorders

Like humans, mice live in complex social groups, fight over territory and mates, and learn when it is safer to avoid certain opponents. After losing even a brief fight, the defeated animals will flee from the mice that hurt them for weeks afterward, a new study shows.

Led by researchers at NYU Grossman School of Medicine, the study, published in Nature, reveals that such "retreating behavior" is influenced by a distinct area on the underside of the hypothalamus, a part of the brain that controls hunger, sleep, and levels of many hormones.

The team had previously found that this special region, called the anterior ventrolateral part of the ventromedial hypothalamus (aVMHvl), helps rodents defend themselves against bullies' attacks. Here, the authors further identified a central role of the area to drive longer-lasting avoidance after being defeated.

The study showed that when rival mice first meet, scent information about opponents is not strong enough to activate aVMHvl cells to prompt a retreat. Once a fight begins, however, pain (such as from getting bitten) triggers the release of the "cuddle hormone" oxytocin.

While this signal has long been linked with parenting and attraction, in this case it binds to oxytocin receptors on aVMHvl cells and signals danger. This process links pain signals to the opponent's scent so the next time the aggressor approaches, its smell alone encourages the bullied mouse to stay away, say the study authors.

"Our findings provide new insight into how oxytocin within the hypothalamus drives learning from traumatic social experiences," said study lead author Takuya Osakada, Ph.D. "While the hormone is often associated with positive behaviors like caregiving, our study highlights its key role in social conflict," adds Osakada, a postdoctoral fellow in the Departments of Psychiatry and Neuroscience and Physiology at NYU Langone Health.

The study team, while cautioning that mice share a lot of brain chemistry with people but are not the same, says previous research has shown similar "retreat" behavior following social defeat in many species including humans. In addition, past studies in children have linked the experience of being bullied to increased social isolation and school absences.

Osakada notes that while previous research had examined rodent behavior over time after experiencing repeated defeats, the new study is the first to explore rapid social learning that occurs immediately after losing a fight.

For the research, the study team observed hundreds of mice that were exposed to a rival for 10 minutes before being separated. They also measured the animals' brain activity before and after a conflict.

The results showed that 24 hours after losing a single fight, social interaction dropped down to just 20% of pre-defeat levels. In addition, the findings revealed that pain prompted the immediate activation of oxytocin-releasing brain cells located right next to the aVMHvl.

To further examine the role of the aVMHvl in social avoidance, the researchers prevented receptors on these cells from binding to oxytocin. They found that rodents with blocked oxytocin receptors were less likely to retreat from their aggressor in later encounters. Meanwhile, when the team instead artificially activated aVMHvl cells, animals kept to themselves even if they had not lost a fight.

"Now that we have a better understanding of critical forces behind social avoidance, researchers can start exploring ways to harness oxytocin to treat disorders that affect social skills, such as autism, social anxiety, and attention-deficit hyperactivity disorder," said study senior author Dayu Lin, Ph.D. Lin is a professor in the Departments of Psychiatry and Neuroscience and Physiology at NYU Langone, as well as a member of its Neuroscience Institute.

That said, Lin cautions that while the team connected the aVMHvl to social avoidance, they found no such link to another behavior exhibited by defeated mice—freezing up in the face of conflict. As a result, researchers say additional brain systems are likely involved in defeat behavior, and understanding such systems is essential before developing oxytocin-based therapies for human social disorders.

The study team next plans to examine whether the newly uncovered aVMHvl mechanism may also be involved in behaviors that rodents use to establish their social hierarchy under more natural conditions, instead of during the contrived scenario from the initial experiment.