Engineers Turn Alexa into an AI Assistant for Tetraplegics

At its core, engineering is about solving problems.

No matter how big a project—whether you need to cross a river, an ocean or interstellar space—engineers are the ones who make it happen. But what if your needs exist on a much smaller scale, such as crossing a room or making a phone call?

For millions of people around the world, the tasks which many of us take for granted are impossible without the use of assistive technologies, such as wheelchairs, hearing aids and prosthetics. As with bridges, ocean liners and interstellar probes, these technologies exist, in large part, through the efforts of engineers.

What sets assistive technologies apart is that their biggest engineering challenges are more often related to cost than to overcoming technical hurdles. Designing and building a motorized wheelchair that responds to voice commands or eye movements is one thing, but it’s quite another to create one that’s affordable for everyone who needs it.

Unless you personally know someone with a disability, it can be easy to overlook this fact.

hackABILITY Projects

That’s part of the raison d'être for hackABILITY, an extended hackathon that seeks to raise awareness about emerging assistive technologies, as well as facilitate the rapid prototyping of new assistive devices. The event was organized by the Mississauga RIC (Research, Innovation and Commercialization) Centre. AIDex was one of seven teams to compete in this year’s hackABILITY, formed with the goal of using AI and speech recognition to help tetraplegics navigate the world around them.

The team even had a specific user in mind.

AIDex team member, Deborah Kennard. (Image courtesy of the AIDex Team.)

“She has an active, intelligent mind and could really use some creative solutions to gain additional control of her life,” wrote AIDex team leader, Tom Burgmann on the engineering.com Projects website. “Our initial goal is to identify and incorporate the AI of verbal speech recognition and build useful functions that convert her spoken commands into actions… The technology components exist but like a PC it needs direction and focus to provide useful results.”

Brainstorming on the engineering.com Projects Whiteboard. (Image courtesy of the AIDex Team.)

Burgmann is an electrical engineer who graduated from MIT in 1981 and has since worked on security systems and therapeutic medical devices for humans and animals. His team included Deborah Kennard—the user for whom the AIDex system was designed—Kit Kennard (her brother), industrial engineering student Nehal Rao and electrical engineer Matt Wong. The team came together via the engineering.com Projects website, which played host to the hackABILITY teams during their development.

The AIDex Wishlist

With a team assembled, the first tasks for AIDex were to define the problem and outline a solution.

“Deborah has the tiniest motion in one finger that allows her to move her wheelchair just slightly, but she’s losing the ability to do that,” explained Burgmann. “So we said, ‘Okay, let’s do a complete, hands-free system.’ Alexa was the easiest toolset—the quickest we could boot up on in such a short period of time.”

CAD model of the enclosure for the AIDex hardware. (Image courtesy of the AIDex team.)

A major motivator for this project was to create a solution that would not only give Deborah greater control over her environment, but to do so at a relatively low cost. Her current system allows her to select from a wheel-based menu, requiring her to wait until the menu rotates to her desired selection. “It’s 30 years old and cost $20,000 at that time,” said Burgmann. “The system hasn’t really been improved, and a replacement is $10,000. We wanted to go from a custom, expensive, service-heavy system to something that’s an off-the-shelf consumer product.”

In light of these considerations, the AIDex team came up with a bill of materials (BOM) for their system, which Burgmann was kind enough to share:

Raspberry Pi 3B
16GB Micro SD Card
Raspberry Pi 3 Enclosure
Dual Microphone Mic-hat from Seeed Tech
External Speaker
5V Power Source/Charger Battery Pack
Custom MicHead Assembly
8 RGB LED Indicator Strip
3D Printed Enclosure with Gooseneck Mounting

Their software needs were met with Raspian Linux, Java, Python and C language modules, as well as a cloud-based IoT hub with analytics on the Microsoft Azure platform. The basis of AIDex’s speech recognition platform came from the Amazon Alexa software development kit (SDK). How the team arrived at Alexa, rather than one of the other speech recognition platforms—such as Apple’s Siri or Microsoft’s Cortana—is a revealing story in its own right.

“We wrote to Apple with a list of our criteria, but their response was that full voice control with Siri is not possible at this time,” said Burgmann. “Plus, Deborah can’t afford to buy a new iPhone. We went to Microsoft, which has an IR camera that can connect to the Fall Edition of Windows 10, but it’s $250. An Echo Dot is $50.”

The team also had difficulty accessing the necessary guides to get up and running on the Azure platform, whereas Amazon offered numerous online guides and examples to help the AIDex team climb the learning curve in just a few short weeks. “There is already a history of many makers who have published their examples of using the Alexa SDK, which provided us complete novices a step stool to creating our first Alexa skill called ‘AIDrian’,” said Burgmann.

Building the AIDex System

Burgmann noted that one of the core functionalities the AIDex team wanted to implement was emergency calling, as a result of their own research into the needs of tetraplegics. “When we looked at Deborah’s other needs,” he said, “we came across a story about a gentleman named Lewis Weelan, who was electrocuted while working on powerlines in Ontario in 2001. He lost the use of three of his limbs and was in a home by himself. When there was a blackout in Ontario in 2003, he died because there was no one to come and help him, because they didn’t know he was in distress.”

The enclosure for the AIDex hardware, 3D printed in ABS. (Image courtesy of the AIDex team.)

To meet this essential need, the AIDex team designed a triply redundant system which is the core of its platform. The first level is based on voice recognition: Deborah can simply ask Alexa to call 911. The second level is based on the system’s ability to recognize when a user’s mouth is open or closed. That was achieved by adding a time-of-flight sensor to the microphone head using LiDAR. If Deborah opens and closes her mouth three times, Alexa will respond by asking, “Do you need emergency help?” If she repeats the action one more time, the system will make an emergency call.

“That’s assuming that Alexa’s fully working,” explained Burgmann. “If it isn’t, we can also go directly through our Bluetooth to a cellphone.” The team also incorporated a flow sensor into Deborah’s water drinking line. “That way,” explained Burgmann, “if she blows on the line instead of sucking, the system can also send an emergency signal. So, that can either be a back-up to the time-of-flight sensor or it can operate concurrently.”

Given the potential for Alexa to malfunction or for Deborah to be unable to communicate her needs in a way the system can recognize, one might wonder why the AIDex team opted for a voice-based system. When we think of tetraplegia, one name that immediately springs to mind is Stephen Hawking, who uses a system that tracks his eye movements.

Electronic components inside the AIDex enclosure. (Image courtesy of the AIDex team.)

When asked whether they considered this method, or even a system based on electroencephalograms (EEGs), Burgmann’s answer once again raised the issue of cost:

“I’ve worked with EEGs, and they’re just not practical,” he said. “I did a project when I was an undergrad using an EEG, and at that time my only option was to hook it into my own scalp with little needles. It was at that point that I thought ‘Maybe this isn’t worth it.’ But, really, we chose Alexa because they had the all the tools available and we had no budget. Our original goal was to do the whole thing for under $1,000, but I think we could do it for well under $200 with low-cost manufacturing techniques.”

Matt Wong, who handled much of the software side of the project, agreed: “The tooling provided by Amazon is relatively complete and easy to use, so the barrier to entry is relatively low.”

What’s Next for AIDex?

Clearly, the principle engineering challenges for AIDex came from the software and integration, rather than hardware. “The control part is not hard,” said Burgmann. “That’s why we went with our own platform, because we have complete control over the I/O using the Raspberry Pi. It’s the voice recognition sending us a signal for what was interpreted that’s the problem. Deborah can say, ‘Turn left,’ for her wheelchair and Alexa will respond ‘Turning left,’ but the Pi doesn’t get a byte that tells it what that message was.”

The AIDex prototype, painted and mounted on a gooseneck test stand. (Image courtesy of the AIDex Team.)

The team is currently working on a solution to this problem that involves sending back an audio tone in a subchannel that the Raspberry Pi can recognize, to give it more direct control in response to voice commands. In addition, in order to minimize the latency of sending voice commands to the cloud for processing, the team has come up with a scheme that uses the wake word engine (e.g., “Alexa”) and keywords coded into the Raspberry Pi, so that the system only calls on cloud-based resources when absolutely necessary.

So, what’s next for the AIDex team? Is this the beginning of a new company?

“I think the possibility is there,” said Burgmann. “We could go from where we are now to a prototype unit for production in about six months.”

To learn more about AIDex and hackABILITY, visit their pages on engineering.com Projects.