Flaws found in video game studies

September 20, 2011 by Lisa Zyga weblog

(Medical Xpress) -- Over the past several years, many studies have found that people who regularly play action video games outperform people who don’t on tasks that involve perception and cognition. However, a new study has found that most, if not all, of these studies suffer from common pitfalls in experimental design, so that the results of the studies are undermined by methodological shortcomings.

The new study notes that, although game training could prove very useful if it works, it is actually in conflict with the typical findings of cognitive training research, which shows that training in one task rarely improves performance in other tasks. In this light, the possibility that video game training causes “broad transfer” of skills beyond gaming itself would be unusual, but potentially very constructive.

In their review article published in Frontiers of Psychology, Walter Boot and Daniel Blakely from Florida State University and Daniel Simons from the University of Illinois have discussed the methodological shortcomings of previous video game studies and offer guidelines for improved testing methods.

First, one of the biggest problems with video game studies was the recruiting process. In many studies, participants knew they were being recruited due to their gaming expertise, which could make them expect to perform well, and the expectation could in turn influence their performance. In contrast, non-gamer participants who were selected without gaming expertise would not experience the same motivation. This recruiting process violates a core principle of experimental design, which Boot, Blakely, and Simons suggest could be improved by covert recruiting. For instance, participants could be asked about their video game experience only at the end of the study or in a prescreening that is not linked to the particular experiment. Still, the review authors note that comparing gamers and non-gamers does not account for the possibility that people may become action gamers because they have the abilities required to excel at these games, or that a third factor might influence both gaming and cognitive abilities.

In other studies, all the participants are non-gamers and they are assigned to either action game training (the experimental group) or non-action game training (the control group). Participants’ performance on tasks involving perception and cognition is measured before and after their training. The problem with this set-up is that participants know which training they’re receiving and whether or not it should improve their performance on the cognitive tasks. Similar to the recruiting problem, participants trained on action video games may expect their performance to improve as a result of their training, while participants in the control group may have no such expectation. Boot, Blakely, and Simons suggest that participants should be asked whether they perceived a connection between their training and the tasks, and also whether they’re aware of past studies that have shown the benefits of gaming on cognitive performance.

The authors of the review also note that the results of some of these studies seem to conflict with one of the most fundamental principles of learning, which is that performance improves with practice. One would expect that who trained on non-action video games should improve at least a bit the second time they perform the cognitive tests, but many studies have found a lack of improvement. This unexpected lack of improvement may give the appearance of a greater benefit of action video game training in comparison. Interestingly, studies that do find an improvement in the control group often find no significant effect of video game training on cognitive performance. As the review authors suggest, in order to make claims about the benefits of video game training, the should perform as expected.

Finally, because video game training studies are costly and time-consuming, the results are often split up and published in multiple journal articles, even though they use the same training groups. Boot, Blakely, and Simons recommend that, when multiple results of a single experiment are reported in different articles, the connection should be clearly stated to avoid confusion and publication bias.

Overall, the review authors emphasize that their criticism does not mean that the results of all previous studies are wrong, just that their experimental methods are not compelling enough to draw strong conclusions. They emphasize that adopting a set of clinical trial best practices could illuminate the true benefits of gaming for cognitive performance. If future studies with improved methods can confirm that action video games do improve perception and cognition, then game training holds great promise for people with vision and attention disorder, as well as for remediating the effects of cognitive aging.

Explore further: Put down that Xbox remote: Researcher suggests video games may not boost cognition

More information: Walter R. Boot, et al. "Do action video games improve perception and cognition?" Frontiers in Psychology. DOI:10.3389/fpsyg.2011.00226

Related Stories

Recommended for you

Study shows there's a positive side to worrying

April 27, 2017

Worry - it does a body good. And, the mind as well. A new paper by Kate Sweeny, psychology professor at the University of California, Riverside, argues there's an upside to worrying.

Study links cannabis use in adolescence to schizophrenia

April 26, 2017

Scientists believe that schizophrenia, a disorder caused by an imbalance in the brain's chemical reactions, is triggered by a genetic interaction with environmental factors. A new Tel Aviv University study published in Human ...


Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Sep 20, 2011
Hrmph, I think the most interesting point raised by this article is the question of whether cognitive skills increase as a result of gaming or do people with better than average cognitive skills gravitate towards gaming more?, I'm a fan of twitch games because I find them engaging after a day of cognitively demanding programming work.
not rated yet Sep 20, 2011
" First, one of the biggest problems with video game studies was the recruiting process. In many studies, participants knew they were being recruited due to their gaming expertise, which could make them expect to perform well, and the expectation could in turn influence their performance. "

They should have taken a clue from the movie " The Last Starfighter ".

Enough from me, I'm off to fight the Ko-Dan Armada..
4 / 5 (1) Sep 21, 2011
I would suspect a common cause for both proficiencies (video games and perception/cognition tasks), probably with a strong neurological basis. An inherent aptitude would undoubtedly encourage the development of such proficiencies.
not rated yet Sep 24, 2011
Kudos Lisa.
Not all is lost - here is your comfort:


Helps establish what is to be understood under 'learning'.
The physical and theoretical basis is well established under a rubric one can label as:
The Nature of Sound.

not rated yet Sep 26, 2011
I also find there is an issue I find when the results of a study is released to the public. With the case of "action video games", you are looking at a very broad range of play styles, interaction, and thinking which is required. Some people adapt to particular playstyles better then others. I have a feeling they are only looking at first person shooter type games, but I canoot tell if they are also looking at real-time stratagies which are totally different.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.