We talk with our hands—and now we know that body language can also be heard

When people move their hands and arms while using their voices, listeners are able to hear it. Even without seeing the messenger, we can pick up each other's body language. Wim Pouw of the Donders Institute of Radboud University has published this finding in PNAS this week.

Most people gesture while talking. From a small wrist movement to complete sign language involving arms, hands and fingers. This form of non-verbal communication supports what someone says, and, in some cases, it is even indispensable in order to explain something properly.

This is difficult if you cannot see the other person, you'd think. But even during phone calls, or when shouting from another room, we often talk with our hands. And that is not pointless at all, according to the study conducted by Wim Pouw and his colleagues from the University of Connecticut: your gestures resonate in your voice.

Louder and higher

It's all about acoustics: the pitch and volume of a voice change together with the movement of arms and hands. "That change is very subtle with a wrist movement," says Pouw. "It is less subtle with an arm movement. The pitch jumps up slightly whenever a movement slows down."

According to Pouw, there are two different causes for these acoustic differences. One of the causes is the creation of vibrations. The forces involved in a movement cause vibrations in your body. Through the connective tissue that holds your body together, vibrations end up in your lungs, affecting the pressure in your lungs.

The second cause is muscle tension around your lungs that is needed to maintain balance. We do not merely use our arm muscles when we move our arms. "When starting the process of stopping your arm from moving, for example, other muscles are suddenly addressed to prevent your body from falling over. These muscles that maintain balance include muscles around your lungs."

Unconsciously hearing an arm movement

As part of this study, Pouw instructed 3 men and 3 women to make a monotone sound, such as 'aaaa," while using all kinds of different hand and arm movements. After that, 30 subjects were asked to listen to the recordings. Not only did they guess which movements were made, but, in many cases, they were also able to mimic these movements simultaneously.

It is easy to measure differences in pitch and volume. It is remarkable that listeners unconsciously identify at what point movement causes these differences. "The subjects not only picked up the speed of the movement, but they also heard the location of the movement."

Emphasize words by hand

Hence, a voice is more than just an abstract collection of sounds, according to the researcher, "When you hear a voice, you literally hear aspects of a person's entire body."

These findings go against the assumption that gestures basically only serve to depict or point out something. "It contributes to the understanding that there is a closer relationship between spoken language and gestures. Hand gestures may have been created to support the voice, to emphasize words, for example."

Google Home

The insights contribute to knowledge about ourselves and to knowledge about speech recognition. Think of systems such as Google Home and Siri. "When developing speech recognition, we have to take movement into account. Think of gestures, but also of someone running while talking, for example."

By means of information about changes in voice, the way how systems can interpret meanings and weights of words can be improved. Or the other way around: noises in a voice produced by a speaker while running, does not affect the meaning of a word. "We can teach these kinds of systems what they should or should not filter out."