Researchers use facial expression recognition technology to develop a game that helps autistic children

May 5, 2014 by Elizabeth Leslie, University of Technology, Sydney
Chek Tien Tan, Natalie Hamblin-Walker, Natalie Harrold and Daniel Rosser. Credit: Joanne Saad.

Autism Spectrum Disorder (ASD) is a lifelong developmental disability that affects, among other things, the way an individual relates to their environment and their interactions with other people. An expression recognition game developed by UTS researchers is helping the emotional development of ASD children – and making it fun.

It's a statistic that's been bandied around for decades – 93 per cent of communication is non-verbal, meaning we rely heavily on our innate ability to read facial and body language cues from others to make sense of the world and build relationships. 

For the one in 100 children with ASD, however, these cues are often missed, making it difficult for them to make friends and engage in social play.

Researchers in the UTS Games Studio have used facial expression to develop CopyMe, a that helps autistic children pick up the cues. Similar 'serious games' usually have an educational or medical objective but rely on the entertainment value to drive the player to achieve those benefits. 

"We wanted CopyMe to be a normal game that anybody can play so that ASD-affected children don't feel like it's assistive technology, and having it designed based on our research, offer some benefits whilst playing in terms of learning about ," says Lecturer in the School of Software Chek Tien Tan. 

Tan supervised Bachelor of Science in Games Development student Natalie Harrold, along with recent graduate and research assistant Daniel Rosser, while they conducted the initial research into human-computer interaction for ASD-affected children.

CopyMe uses the facial expression recognition technology to teach children about emotions by having them progress through the game by mimicking and repeating six main facial expressions: happy, sad, scared, surprised, angry and disgusted.

This repeated copying of facial expressions has been previously shown to help children on the autism spectrum recognise emotions in others, which may help them improve how they interact with other children. 

CopyMe. Credit: Natalie Harrold.

"The game uses deformable model fitting, which basically means it applies a 3D model to track your face, using open-sourced computer vision technology," says Rosser. 

Using a set of faces and expressions as a template, the team enhanced this and ran it through nine months of testing and development. 

Being one of the first projects to use real-time facial expression recognition on the iPad, Harrold says it was trial and error in the early stages.

"We spent a lot of time testing and doing trials in the Games Studio; testing the technology on our own faces, improving the sensitivity of the face tracker and the lag. My two-year-old niece understood the aim of the game and wanted to play it, but we uncovered new hurdles with her very small face and pale eyebrows – the facial tracker had issues 'locking on' for her." 

The team also tested the app in an after-school care centre in Sydney with a mixed group of children. They found the ASD children not only enjoyed playing the game, but also succeeded in mastering several of the levels. 

"The idea is to make it feel like a fun game rather than just a task that has to be completed over and over," explains Harrold. 

Being iPad-based also removes the stigma attached to specialised ASD intervention tools, along with the need for adult supervision. Once the game is set up, the child can play in a self-directed way, with carers able to monitor improvement through high scores.  

"We'd read so many papers suggesting the first big hurdle is just getting them to interact with an intervention, but the fact that it was on the iPad made it easier," says Harrold. 

Feedback from centre staff has been positive following the trial. Natalie Hamblin-Walker works one-on-one with the children and was amazed to see the impact playing the game had on the autism-affected children in her care. 

"The non-ASD-affected kids thought the game was good, but needed more elements and harder levels, " says Hamblin-Walker. "However, the two ASD-affected kids were hooked. They didn't want to give the iPad back and they even got all the faces right, which was incredible." 

Although it was a small trial, the anecdotal results are encouraging enough to confirm for the team that CopyMe meets the parameters required for a specialised ASD program. "It was really nice to hear feedback that the kids with autism could complete the levels," says Tan. "We actually weren't sure what to expect."

With the app having already been downloaded over 700 times in its first week, the team hope to gather feedback from an in-app survey for suggested further improvements to the game. "We'd like to be able to integrate gameplay that demonstrates when those emotions are shown and add mini-challenges and more levels," says Rosser.

Along with undertaking a broader trial involving more , the team intend to open source the project so other researchers and game developers can expand the technology. A set of guidelines have already been published by the team for game developers of specialised ASD programs, covering advice on colours, sounds, even the size of screen buttons required for the on-screen elements.

"Since this is one of the first projects to use real-time facial recognition on the iPad, people would be trying to figure out how we did it," Tan says. "As a researcher, it's important to share this and allow others to take it further."

Explore further: Children and teens with autism more likely to become preoccupied with video games

Related Stories

Children and teens with autism more likely to become preoccupied with video games

April 17, 2013
Children and teens with autism spectrum disorder (ASD) use screen-based media, such as television and video games, more often than their typically developing peers and are more likely to develop problematic video game habits, ...

New software concept may help with autism

July 5, 2013
Assistive learning software to help autistic children recognise emotions from facial expressions recently won a UniServices prize in the annual Spark Ideas Challenge.

Robots help teach social skills to children

April 28, 2014
(Medical Xpress)—Robots and humans socialize frequently in pop fiction—think of Wall-E and Star Trek: The Next Generation. Now, a UT Dallas researcher is giving the fantasy of robotic friends a practical edge with a robot ...

Speech disrupts facial attention in six-month-olds who later develop autism

February 4, 2014
From birth, infants naturally show a preference for human contact and interaction, including faces and voices. These basic predispositions to social stimuli are altered in individuals diagnosed with autism spectrum disorders ...

New method of screening children for autism spectrum disorders works at nine months old

April 15, 2014
Researchers, including a team from Children's National Health System, have identified head circumference and head tilting reflex as two reliable biomarkers in the identification of autism spectrum disorders (ASD) in children ...

At 6 months, development of children with autism like those without

October 30, 2012
The development of children with Autism Spectrum Disorder (ASD) is much like that of children without ASD at 6 months of age, but differs afterwards. That's the main finding of the largest prospective, longitudinal study ...

Recommended for you

Nearly imperceptible fluctuations in movement correspond to autism diagnoses

January 17, 2018
A new study led by researchers at Indiana University and Rutgers University provides the strongest evidence yet that nearly imperceptible changes in how people move can be used to diagnose neurodevelopmental disorders, including ...

Epigenetics study helps focus search for autism risk factors

January 16, 2018
Scientists have long tried to pin down the causes of autism spectrum disorder. Recent studies have expanded the search for genetic links from identifying genes toward epigenetics, the study of factors that control gene expression ...

Being bilingual may help autistic children

January 16, 2018
Children with Autism Spectrum Disorders (ASD) often have a hard time switching gears from one task to another. But being bilingual may actually make it a bit easier for them to do so, according to a new study which was recently ...

No rise in autism in US in past three years: study

January 2, 2018
After more than a decade of steady increases in the rate of children diagnosed with autism in the United States, the rate has plateaued in the past three years, researchers said Tuesday.

Autism therapy: Brain stimulation restores social behavior in mice

December 13, 2017
Scientists are examining the feasibility of treating autistic children with neuromodulation after a new study showed social impairments can be corrected by brain stimulation.

Social phobia linked to autism and schizophrenia

December 11, 2017
New Swinburne research shows that people who find social situations difficult tend to have similar brain responses to those with schizophrenia or autism.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.