Researchers use facial expression recognition technology to develop a game that helps autistic children

Autism Spectrum Disorder (ASD) is a lifelong developmental disability that affects, among other things, the way an individual relates to their environment and their interactions with other people. An expression recognition game developed by UTS researchers is helping the emotional development of ASD children – and making it fun.

It's a statistic that's been bandied around for decades – 93 per cent of communication is non-verbal, meaning we rely heavily on our innate ability to read facial and body language cues from others to make sense of the world and build relationships. 

For the one in 100 children with ASD, however, these cues are often missed, making it difficult for them to make friends and engage in social play.

Researchers in the UTS Games Studio have used facial expression recognition technology to develop CopyMe, a game that helps autistic children pick up the cues. Similar 'serious games' usually have an educational or medical objective but rely on the entertainment value to drive the player to achieve those benefits. 

"We wanted CopyMe to be a normal game that anybody can play so that ASD-affected children don't feel like it's assistive technology, and having it designed based on our research, offer some benefits whilst playing in terms of learning about facial expressions," says Lecturer in the School of Software Chek Tien Tan. 

Tan supervised Bachelor of Science in Games Development student Natalie Harrold, along with recent graduate and research assistant Daniel Rosser, while they conducted the initial research into human-computer interaction for ASD-affected children.

CopyMe uses the facial expression recognition technology to teach children about emotions by having them progress through the game by mimicking and repeating six main facial expressions: happy, sad, scared, surprised, angry and disgusted.

This repeated copying of facial expressions has been previously shown to help children on the autism spectrum recognise emotions in others, which may help them improve how they interact with other children. 

"The game uses deformable model fitting, which basically means it applies a 3D model to track your face, using open-sourced computer vision technology," says Rosser. 

Using a set of faces and expressions as a template, the team enhanced this and ran it through nine months of testing and development. 

Being one of the first projects to use real-time facial expression recognition on the iPad, Harrold says it was trial and error in the early stages.

"We spent a lot of time testing and doing trials in the Games Studio; testing the technology on our own faces, improving the sensitivity of the face tracker and the lag. My two-year-old niece understood the aim of the game and wanted to play it, but we uncovered new hurdles with her very small face and pale eyebrows – the facial tracker had issues 'locking on' for her." 

The team also tested the app in an after-school care centre in Sydney with a mixed group of children. They found the ASD children not only enjoyed playing the game, but also succeeded in mastering several of the levels. 

"The idea is to make it feel like a fun game rather than just a task that has to be completed over and over," explains Harrold. 

Being iPad-based also removes the stigma attached to specialised ASD intervention tools, along with the need for adult supervision. Once the game is set up, the child can play in a self-directed way, with carers able to monitor improvement through high scores.  

"We'd read so many papers suggesting the first big hurdle is just getting them to interact with an intervention, but the fact that it was on the iPad made it easier," says Harrold. 

Feedback from centre staff has been positive following the trial. Natalie Hamblin-Walker works one-on-one with the children and was amazed to see the impact playing the game had on the autism-affected children in her care. 

"The non-ASD-affected kids thought the game was good, but needed more elements and harder levels, " says Hamblin-Walker. "However, the two ASD-affected kids were hooked. They didn't want to give the iPad back and they even got all the faces right, which was incredible." 

Although it was a small trial, the anecdotal results are encouraging enough to confirm for the team that CopyMe meets the parameters required for a specialised ASD program. "It was really nice to hear feedback that the kids with autism could complete the levels," says Tan. "We actually weren't sure what to expect."

With the app having already been downloaded over 700 times in its first week, the team hope to gather feedback from an in-app survey for suggested further improvements to the game. "We'd like to be able to integrate gameplay that demonstrates when those emotions are shown and add mini-challenges and more levels," says Rosser.

Along with undertaking a broader trial involving more children, the team intend to open source the project so other researchers and game developers can expand the technology. A set of guidelines have already been published by the team for game developers of specialised ASD programs, covering advice on colours, sounds, even the size of screen buttons required for the on-screen elements.

"Since this is one of the first projects to use real-time facial recognition on the iPad, people would be trying to figure out how we did it," Tan says. "As a researcher, it's important to share this and allow others to take it further."