Flaws found in video game studies

September 20, 2011 by Lisa Zyga weblog

(Medical Xpress) -- Over the past several years, many studies have found that people who regularly play action video games outperform people who don’t on tasks that involve perception and cognition. However, a new study has found that most, if not all, of these studies suffer from common pitfalls in experimental design, so that the results of the studies are undermined by methodological shortcomings.

The new study notes that, although game training could prove very useful if it works, it is actually in conflict with the typical findings of cognitive training research, which shows that training in one task rarely improves performance in other tasks. In this light, the possibility that video game training causes “broad transfer” of skills beyond gaming itself would be unusual, but potentially very constructive.

In their review article published in Frontiers of Psychology, Walter Boot and Daniel Blakely from Florida State University and Daniel Simons from the University of Illinois have discussed the methodological shortcomings of previous video game studies and offer guidelines for improved testing methods.

First, one of the biggest problems with video game studies was the recruiting process. In many studies, participants knew they were being recruited due to their gaming expertise, which could make them expect to perform well, and the expectation could in turn influence their performance. In contrast, non-gamer participants who were selected without gaming expertise would not experience the same motivation. This recruiting process violates a core principle of experimental design, which Boot, Blakely, and Simons suggest could be improved by covert recruiting. For instance, participants could be asked about their video game experience only at the end of the study or in a prescreening that is not linked to the particular experiment. Still, the review authors note that comparing gamers and non-gamers does not account for the possibility that people may become action gamers because they have the abilities required to excel at these games, or that a third factor might influence both gaming and cognitive abilities.

In other studies, all the participants are non-gamers and they are assigned to either action game training (the experimental group) or non-action game training (the control group). Participants’ performance on tasks involving perception and cognition is measured before and after their training. The problem with this set-up is that participants know which training they’re receiving and whether or not it should improve their performance on the cognitive tasks. Similar to the recruiting problem, participants trained on action video games may expect their performance to improve as a result of their training, while participants in the control group may have no such expectation. Boot, Blakely, and Simons suggest that participants should be asked whether they perceived a connection between their training and the tasks, and also whether they’re aware of past studies that have shown the benefits of gaming on cognitive performance.

The authors of the review also note that the results of some of these studies seem to conflict with one of the most fundamental principles of learning, which is that performance improves with practice. One would expect that who trained on non-action video games should improve at least a bit the second time they perform the cognitive tests, but many studies have found a lack of improvement. This unexpected lack of improvement may give the appearance of a greater benefit of action video game training in comparison. Interestingly, studies that do find an improvement in the control group often find no significant effect of video game training on cognitive performance. As the review authors suggest, in order to make claims about the benefits of video game training, the should perform as expected.

Finally, because video game training studies are costly and time-consuming, the results are often split up and published in multiple journal articles, even though they use the same training groups. Boot, Blakely, and Simons recommend that, when multiple results of a single experiment are reported in different articles, the connection should be clearly stated to avoid confusion and publication bias.

Overall, the review authors emphasize that their criticism does not mean that the results of all previous studies are wrong, just that their experimental methods are not compelling enough to draw strong conclusions. They emphasize that adopting a set of clinical trial best practices could illuminate the true benefits of gaming for cognitive performance. If future studies with improved methods can confirm that action video games do improve perception and cognition, then game training holds great promise for people with vision and attention disorder, as well as for remediating the effects of cognitive aging.

Explore further: Video games shown to improve vision

More information: Walter R. Boot, et al. "Do action video games improve perception and cognition?" Frontiers in Psychology. DOI:10.3389/fpsyg.2011.00226

Related Stories

Video games shown to improve vision

March 15, 2007

According to a new study from the University of Rochester, playing action video games sharpens vision. In tests of visual acuity that assess the ability to see objects accurately in a cluttered space, game players scored ...

Action video games improve vision

March 29, 2009

Video games that involve high levels of action, such as first-person-shooter games, increase a player's real-world vision, according to research in today's Nature Neuroscience.

Racing, shooting and zapping your way to better visual skills

December 17, 2009

Do your kids want a Wii, a PlayStation or an Xbox 360 this year? This holiday gift season is packed with popular gaming systems and adrenaline-pumping, sharpshooting games. What's a parent to do? Is there any redeeming value ...

Video games lead to faster decisions that are no less accurate

September 13, 2010

Cognitive scientists from the University of Rochester have discovered that playing action video games trains people to make the right decisions faster. The researchers found that video game players develop a heightened sensitivity ...

How video games stretch the limits of our visual attention

November 18, 2010

They are often accused of being distracting, but recent research has found that action packed video games like Halo and Call of Duty can enhance visual attention, the ability that allows us to focus on relevant visual information. ...

Recommended for you

Why we fall prey to misinformation

August 23, 2016

Even when we know better, we often rely on inaccurate or misleading information to make future decisions. But why are we so easily influenced by false statements such as "vaccinations cause autism" or "30 million illegal ...

Sleep makes relearning faster and longer-lasting

August 22, 2016

Getting some sleep in between study sessions may make it easier to recall what you studied and relearn what you've forgotten, even 6 months later, according to new findings from Psychological Science, a journal of the Association ...

Have we misunderstood post-traumatic stress disorder?

August 22, 2016

In understanding war-related post-traumatic stress disorder, a person's cultural and professional context is just as important as how they cope with witnessing wartime events, which could change the way mental health experts ...


Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Sep 20, 2011
Hrmph, I think the most interesting point raised by this article is the question of whether cognitive skills increase as a result of gaming or do people with better than average cognitive skills gravitate towards gaming more?, I'm a fan of twitch games because I find them engaging after a day of cognitively demanding programming work.
not rated yet Sep 20, 2011
" First, one of the biggest problems with video game studies was the recruiting process. In many studies, participants knew they were being recruited due to their gaming expertise, which could make them expect to perform well, and the expectation could in turn influence their performance. "

They should have taken a clue from the movie " The Last Starfighter ".

Enough from me, I'm off to fight the Ko-Dan Armada..
4 / 5 (1) Sep 21, 2011
I would suspect a common cause for both proficiencies (video games and perception/cognition tasks), probably with a strong neurological basis. An inherent aptitude would undoubtedly encourage the development of such proficiencies.
not rated yet Sep 24, 2011
Kudos Lisa.
Not all is lost - here is your comfort:


Helps establish what is to be understood under 'learning'.
The physical and theoretical basis is well established under a rubric one can label as:
The Nature of Sound.

not rated yet Sep 26, 2011
I also find there is an issue I find when the results of a study is released to the public. With the case of "action video games", you are looking at a very broad range of play styles, interaction, and thinking which is required. Some people adapt to particular playstyles better then others. I have a feeling they are only looking at first person shooter type games, but I canoot tell if they are also looking at real-time stratagies which are totally different.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.