Building a storytelling app to support literacy amongst deaf children

Advances in digital technology including tablets, smartphones, and mobile apps have revolutionized the way people interact, communicate, and learn at home (and in classrooms) around the world. Now, an international team of researchers are working to ensure that this shifting educational landscape doesn't leave deaf children behind and will help support literacy efforts throughout the community. The project, in Telematics and Informatics, published by Elsevier, was recently selected by an external advisory board made up of individuals from NGOs to receive the Atlas Award.

"We're really proposing a framework that uses storytelling (and digital technology) as a way to motivate children to learn," said Leandro Flórez-Aristizábal, Ph.D. candidate at the Institución Universitaria Antonio José Camacho and University of Cauca, Colombia.

Flórez-Aristizábal and her colleagues are currently working to design and test an interactive storytelling app for sign language-speaking deaf children. To help in this effort, they've enlisted the experts: deaf children themselves. By first designing a series of black and white cards depicting images, Flórez-Aristizábal's research team developed a general storyline, but the sequence of events was up to the kids to decide.

This interactive storytelling approach was tested as a series of case studies, involving children between the ages of 9-and-13-years at the Institute of Special Therapy of Senses in Cali, Colombia. A single student would arrange the cards in any order they chose and tell the story to the rest of their class through sign language.

The researchers then added additional details and colors to the cards for further testing. Eventually the colorful cards were converted to a digital format for use with a tablet or personal computer. The initial findings found that a child's motivation increased when technology was part of the activity, specifically when rewards in the form of virtual trophies were offered.

"We started with deafness, but we don't want to stop there," said Mr. Flórez-Aristizábal. "We want to invite other researchers to join us in this journey and help us develop more prototypes, and adaptations for other disabilities and other educational goals. In this way we're trying to fill the gap. Ultimately, we want to forget about disabilities and think about different abilities."

Flórez-Aristizábal added that the aim is to continue working, and ultimately, develop an app with multiple levels and support multiple sign-languages, including Spanish, British and Colombian Sign Language.

A Conversation with Leandro Flórez-Aristizábal

Atlas: How did you begin thinking about digital transformation and what it means for the deaf community?

Flórez-Aristizábal: This is part of my PhD thesis. When I started, I was continuing work of my supervisor with deaf children who used cochlear implant. I wanted to continue this work by including children who use sign language. In this paper, we have done some case studies with deaf children—trying to understand how they learn, what they like, the stories they enjoy. We came up with the idea of involving children in the design of a prototype system but also in designing the story behind the idea. We want to support teachers in doing their job, but also to provide children with something they can enjoy to promote literacy.

Atlas: What did you learn from the kids?

Flórez-Aristizábal: We wanted to know what they thought about the story and the drawings, and it was interesting. We as adults didn't care much about some details that they paid attention to. For instance, one of the characters was supposed to be surprised, but the kids thought he was scared because of the way the eyes were drawn. It's something we knew we had to take into account and that's why it's important to have a paper prototype first. In the next case study, we already implemented the kids' suggestions about the drawings, the facial expressions, and so on.

Atlas: How did things change when you moved to the digital version?

Flórez-Aristizábal: For the 3rd case study, we had a digital version of the story and this is something they actually enjoy much more. They could arrange the cards [on the screen] how they wanted and they didn't want to stop. They wanted to continue working with the computer. We added something we didn't have in the previous case studies including some gamification aspects. After they finished arranging, we showed them their time and a trophy like a reward. They immediately understood and tried to arrange cards over and over. They wanted to beat their previous times. We knew it was a way to engage children through rewards and gamification. But it's something you should do carefully to ensure children don't lose focus on the purpose of the activity.

Atlas: This looks promising, so what are you doing now?

Flórez-Aristizábal:  We recently did our 5th case study with Scottish deaf children. We tested a prototype mobile app with storytelling and a vocabulary stage. In the end, they had to play a game about matching words with signs and images and they really enjoyed it. We're now ready to have the framework evaluated by some experts. The pedagogical aspect has to be evaluated by teachers for it's potential for not just storytelling but also to promote collaborative learning among deaf children.

Atlas: Was there anything that particularly struck you in watching these kids interact?

Flórez-Aristizábal: In this last case study we did with deaf children in Scotland, I was surprised how well they interact with mobile devices. It's natural for them. They immediately knew how to interact with the app. I thought they'd need an introduction or help, but no, they just did it from the beginning to the end of the activity. They didn't need any help and they really enjoyed it. We are going to test the same prototypes in Colombia and see how differences in digital literacy also may influence this. I think it could give us different insights about how children use technology in different countries. Many deaf children in Colombia are from low income families and don't have access to this technology at home or in school. So, we'll see how they will interact with the app.

Atlas: Are there plans to make such an app available?

Flórez-Aristizábal: Well, yes, so far we've done prototypes. This one we just tested is supposed to have nine levels, but so far we have one. All we wanted to know is how children interact with it. The idea is to have a full version of the app with different levels and languages by next year and to make it available in the app store. But we don't want to stop there. We want to focus on other skills like math or perhaps children with other disabilities like blindness or cognitive disabilities.

We've been working on this for the last four years with researchers from Saudi Arabia, Portugal, Columbia, and Spain. It's very rewarding to do this kind of research. It's not something that you just type or write in a paper and it will be there in a database nobody uses. We want to do something that can impact on people's life and this is why we are going to focus our research on disabilities.