Swinburne graduates develop software that trained over 2000 carers
- This article originally featured in Swinburne’s Venture magazine.
Imagine a world where dark holes appear in the floor, you can’t tell the toilet from the bathroom sink, and the sudden whir of a fridge motor makes you think someone is breaking into your home.
This is the nightmarish reality for Australians facing the dual challenge of ageing and dementia.
“Age–related eye conditions can affect the way people perceive objects,” says Dr Tanya Petrovich, manager of business development at Alzheimer’s Australia Victoria. “Dementia can affect neurons in the brain. They form tangles and become surrounded by plaque. They don’t communicate properly – and when neurons don’t make connections, they die.”
The combination of failing eyesight and dying neurons means people with dementia interpret the world around them very differently. “For example, they may no longer make a connection between a once–familiar noise and what it stood for,” Dr Petrovich says.
“Most people would hear a fridge motor start, and, from memory, associate it with a fridge. A person with dementia might make the wrong connection, and believe someone is breaking in.”
Dementia is the greatest cause of disability in Australians aged over 65 and affects more than 340,000 people nationwide. It’s a growing community concern, and Swinburne University of Technology teams have been at the forefront of research assisting carers to understand the skewed and difficult reality that people with the condition face.
Opaque Multimedia, founded by four Swinburne graduates in 2008, was commissioned by Alzheimer’s Australia Vic to develop a technology that allows carers to enter the “virtual worlds” of people living with dementia.
Norman Wang, James Bonner, Chris Mackenzie and Liam McGuire developed the Virtual Dementia ExperienceTM (VDETM), which has won multiple awards and been used to train more than 2000 professional carers.
VDE uses Unreal Engine gaming software. “It’s an interactive gaming technology that underlies some of the biggest–selling games in the world,” Mr Wang says.
‘Kinecting’ with a different reality
The team chose Microsoft’s Kinect as the user interface. “We knew that people in their 40s and 50s were the primary carer demographic and would make up most of the people being trained,” Mr Wang says. “It was important to use a ‘natural’ interface instead of a keypad or gamepad.”
The Kinect interface is a touchless sensor that studies and mirrors a person’s motion. It creates a series of virtual “skeletons” from which software developers can create meaningful input.
“If you want to open a door, you just reach out to the handle,” Mr Wang says. “A virtual avatar that mirrors your motion will reach out and open the handle.”
Verifiably better learning
Alzheimer’s Australia Vic also engaged a Swinburne team, led by Dr Wendy Doube and Dr Sunil Bhar, to gauge the effectiveness of VDE workshops. “The team found that VDE learning is significantly more likely than equivalent traditional classroom training to create an empathic understanding of how people with dementia think and feel,” Dr Petrovich says.
“Anecdotally, we know the VDE is opening carers’ minds to the idea that they can be change agents – they don’t have to be the CEO to make things happen.”
Immersion in a frightening world
During a typical training session with the program, a facilitator leads users through simulated scenarios, using a tablet or other mobile device. “We’ve based the scenarios on true stories from carers,” Dr Petrovich says.
“In one case, a person with dementia went to brush their teeth, but picked up a razor instead. The items are similar in size, have heads, and are kept in the same place. The person knew roughly what a toothbrush should look like, but could no longer discern it from similar objects.”
The VDE leads carers through many other challenges to bring the confusion of dementia alive. “As the player walks down the hallway, their vision is blurred and the patterned wallpaper seems to move,” she says. “The white–on–white bathroom reflects glare and makes it hard to distinguish the toilet from the hand basin, while the dark mat on the bathroom floor appears to turn into a dangerous hole.”
The virtual reality program is based at the Perc Walkley Dementia Learning Centre in the Melbourne suburb of Parkville. “One of the things we teach professional carers is that as a person’s dementia progresses, the best way to communicate with them is through the senses,” Dr Petrovich says.
“Many people have difficulty hearing as they get older, but someone with dementia can lose the ability to distinguish an individual conversation from background noise. If there’s a vacuum cleaner running or a lot of noise from the kitchen, a person with dementia may have real difficulty understanding you. This can make them upset or aggressive.
“Carers who’ve trained with the VDE say they’re now making living environments quieter for people with dementia, and have noticed changes in their behaviour.”
Last August, Opaque Multimedia won the world citizenship category of the prestigious Microsoft Imagine Cup for its work on the VDE.
Winning the award has enabled the team to connect with people at all levels of Microsoft, including CEO Satya Nadella. The prize includes a trip to Seattle, scheduled for January 2016. Microsoft might also help with the program’s future development.
Opaque Multimedia now has about dozen employees. Its clients include Google, NASA and Lucasfilm.
Into the future
The next frontier in dementia–friendly environments is an interactive game designed for people with dementia. The Forest, developed by Alzheimer’s Australia Vic, assisted by Opaque Multimedia and sponsored by Lifeview Residential Care, is slated for release in April 2016.
The Forest aims to address a key issue of dementia: disengagement. “As the disease progresses, people may lose certain skills, such as knitting or sewing,” Dr Petrovich says. “It also gets harder to concentrate for long, so people can become withdrawn from activities they used to enjoy.”
The technology uses a serene–yet–interactive environment with moving animals, greenery and brightly coloured flowers, to engage people in the middle and later stages of dementia in something beyond themselves, and give them back some control.
“As dementia progresses, other people make a lot of choices for you,” Dr Petrovich says. “In The Forest, people can make choices about what happens, which creates a sense of empowerment.
“It’s the first time anything like this has been done,” she says. “It could go anywhere.”
Without a medical breakthrough, it is expected there will be 900,000 Australians with dementia by 2050. Given the array of challenges that arise from ageing and dementia, creating accessible technology will become increasingly important.
Software engineer Rohan Liston from the Swinburne Software Innovation Lab has worked with Alzheimer’s Australia Vic to develop the Dementia Friendly Home app for Apple and Android devices.
Like the virtual reality program, the app is underpinned by Unreal Engine. “Almost every object
in the virtual home is interactive,” Mr Liston says. “Carers and families can tap and swipe their way through the rooms, turn the dementia–friendly features on and off, and view suggestions for making a real–life home safer and more liveable.”
The technology was based on work by Kirsty Bennett and Richard Fleming from the University of Wollongong. They developed 10 Dementia Enabling Environment Principles, which include risk reduction, ease of visibility and appropriate levels of stimulation.
“We wanted carers to engage with these principles more deeply, to enrich their meaning beyond words on a page and put them into a life–like context,” Dr Petrovich says.
The app should be available from the Apple and Android app stores for a few dollars before the end
of 2015. Proceeds will go to Alzheimer’s Australia Vic, to help support its work.
Media enquiries0455 502 999