Researchers use facial expression recognition technology to develop a game that helps autistic children

May 5, 2014 by Elizabeth Leslie
Chek Tien Tan, Natalie Hamblin-Walker, Natalie Harrold and Daniel Rosser. Credit: Joanne Saad.

Autism Spectrum Disorder (ASD) is a lifelong developmental disability that affects, among other things, the way an individual relates to their environment and their interactions with other people. An expression recognition game developed by UTS researchers is helping the emotional development of ASD children – and making it fun.

It's a statistic that's been bandied around for decades – 93 per cent of communication is non-verbal, meaning we rely heavily on our innate ability to read facial and body language cues from others to make sense of the world and build relationships. 

For the one in 100 children with ASD, however, these cues are often missed, making it difficult for them to make friends and engage in social play.

Researchers in the UTS Games Studio have used facial expression to develop CopyMe, a that helps autistic children pick up the cues. Similar 'serious games' usually have an educational or medical objective but rely on the entertainment value to drive the player to achieve those benefits. 

"We wanted CopyMe to be a normal game that anybody can play so that ASD-affected children don't feel like it's assistive technology, and having it designed based on our research, offer some benefits whilst playing in terms of learning about ," says Lecturer in the School of Software Chek Tien Tan. 

Tan supervised Bachelor of Science in Games Development student Natalie Harrold, along with recent graduate and research assistant Daniel Rosser, while they conducted the initial research into human-computer interaction for ASD-affected children.

CopyMe uses the facial expression recognition technology to teach children about emotions by having them progress through the game by mimicking and repeating six main facial expressions: happy, sad, scared, surprised, angry and disgusted.

This repeated copying of facial expressions has been previously shown to help children on the autism spectrum recognise emotions in others, which may help them improve how they interact with other children. 

CopyMe. Credit: Natalie Harrold.

"The game uses deformable model fitting, which basically means it applies a 3D model to track your face, using open-sourced computer vision technology," says Rosser. 

Using a set of faces and expressions as a template, the team enhanced this and ran it through nine months of testing and development. 

Being one of the first projects to use real-time facial expression recognition on the iPad, Harrold says it was trial and error in the early stages.

"We spent a lot of time testing and doing trials in the Games Studio; testing the technology on our own faces, improving the sensitivity of the face tracker and the lag. My two-year-old niece understood the aim of the game and wanted to play it, but we uncovered new hurdles with her very small face and pale eyebrows – the facial tracker had issues 'locking on' for her." 

The team also tested the app in an after-school care centre in Sydney with a mixed group of children. They found the ASD children not only enjoyed playing the game, but also succeeded in mastering several of the levels. 

"The idea is to make it feel like a fun game rather than just a task that has to be completed over and over," explains Harrold. 

Being iPad-based also removes the stigma attached to specialised ASD intervention tools, along with the need for adult supervision. Once the game is set up, the child can play in a self-directed way, with carers able to monitor improvement through high scores.  

"We'd read so many papers suggesting the first big hurdle is just getting them to interact with an intervention, but the fact that it was on the iPad made it easier," says Harrold. 

The video will load shortly

Feedback from centre staff has been positive following the trial. Natalie Hamblin-Walker works one-on-one with the children and was amazed to see the impact playing the game had on the autism-affected children in her care. 

"The non-ASD-affected kids thought the game was good, but needed more elements and harder levels, " says Hamblin-Walker. "However, the two ASD-affected kids were hooked. They didn't want to give the iPad back and they even got all the faces right, which was incredible." 

Although it was a small trial, the anecdotal results are encouraging enough to confirm for the team that CopyMe meets the parameters required for a specialised ASD program. "It was really nice to hear feedback that the kids with autism could complete the levels," says Tan. "We actually weren't sure what to expect."

With the app having already been downloaded over 700 times in its first week, the team hope to gather feedback from an in-app survey for suggested further improvements to the game. "We'd like to be able to integrate gameplay that demonstrates when those emotions are shown and add mini-challenges and more levels," says Rosser.

Along with undertaking a broader trial involving more , the team intend to open source the project so other researchers and game developers can expand the technology. A set of guidelines have already been published by the team for game developers of specialised ASD programs, covering advice on colours, sounds, even the size of screen buttons required for the on-screen elements.

"Since this is one of the first projects to use real-time facial recognition on the iPad, people would be trying to figure out how we did it," Tan says. "As a researcher, it's important to share this and allow others to take it further."

Explore further: Children and teens with autism more likely to become preoccupied with video games

Related Stories

Children and teens with autism more likely to become preoccupied with video games

April 17, 2013
Children and teens with autism spectrum disorder (ASD) use screen-based media, such as television and video games, more often than their typically developing peers and are more likely to develop problematic video game habits, ...

New software concept may help with autism

July 5, 2013
Assistive learning software to help autistic children recognise emotions from facial expressions recently won a UniServices prize in the annual Spark Ideas Challenge.

Robots help teach social skills to children

April 28, 2014
(Medical Xpress)—Robots and humans socialize frequently in pop fiction—think of Wall-E and Star Trek: The Next Generation. Now, a UT Dallas researcher is giving the fantasy of robotic friends a practical edge with a robot ...

Speech disrupts facial attention in six-month-olds who later develop autism

February 4, 2014
From birth, infants naturally show a preference for human contact and interaction, including faces and voices. These basic predispositions to social stimuli are altered in individuals diagnosed with autism spectrum disorders ...

New method of screening children for autism spectrum disorders works at nine months old

April 15, 2014
Researchers, including a team from Children's National Health System, have identified head circumference and head tilting reflex as two reliable biomarkers in the identification of autism spectrum disorders (ASD) in children ...

At 6 months, development of children with autism like those without

October 30, 2012
The development of children with Autism Spectrum Disorder (ASD) is much like that of children without ASD at 6 months of age, but differs afterwards. That's the main finding of the largest prospective, longitudinal study ...

Recommended for you

Late-breaking mutations may play an important role in autism

July 17, 2017
A study of nearly 6,000 families, combining three genetic sequencing technologies, finds that mutations that occur after conception play an important role in autism. A team led by investigators at Boston Children's Hospital ...

Females with autism show greater difficulty with day-to-day tasks than male counterparts

July 14, 2017
Women and girls with autism may face greater challenges with real world planning, organization and other daily living skills, according to a study published in the journal Autism Research.

Researchers investigate possible link between carnitine deficiency and autism

July 13, 2017
Researchers are always looking for new clues to the causes of autism, with special emphasis on prevention or treatment. At Baylor College of Medicine, Dr. Arthur Beaudet has been following clinical and genetic clues in patients ...

How children look at mom's face is influenced by genetic factors and altered in autism

July 12, 2017
New research has uncovered compelling evidence that genetics plays a major role in how children look at the world and whether they have a preference for gazing at people's eyes and faces or at objects.

Oxytocin improves social abilities in some kids with autism, study finds

July 10, 2017
Children with autism showed improved social behavior when treated with oxytocin, a hormone linked to social abilities, according to a new study by researchers at the Stanford University School of Medicine. Children with low ...

Possible early diagnosis of autism spectrum disorder

June 29, 2017
Measuring a set of proteins in the blood may enable earlier diagnosis of autism spectrum disorder (ASD), according to a study from the Peter O'Donnell Jr. Brain Institute at UT Southwestern Medical Center.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.