Future of immersive gaming gear for the blind

Researchers explore how their real-world assistive technology could bring better, augmented gaming experiences to people who can't see.

Today's biggest games, like the Witcher 3 and Metal Gear Solid 5, are graphical masterpieces, set in immense worlds where every leaf and pebble is computationally rendered into lifelike detail.

But these worlds aren't so astonishing for the nearly 40 million blind people and 250 million people with impaired vision living in the world today.

Unless games are redesigned as "video-less" experiences, where players rely on aural or haptic senses to navigate through levels, it's nearly impossible for blind people to explore truly immersive video games.

Touchscreens, gyroscopes and haptic vibration technologies are helping blind players navigate many games today, but a small group of researchers at Intel believe their real-world assistive technology, based on Intel RealSense cameras and wearable sensors, could bring virtual and augmented reality-infused games to people who can't see.

"There's been a lot of work done on video games for blind people in the last 10 or 20 years with sound," said Chandrika "Shani" Jayant, a senior user experience designer at Intel's RealSense Interaction Design Group.

"Today, with augmented and virtual reality, you could take in things from your real world, like the shape of the room you're sitting in or the people around you, and use those things to actually change what happens in the game that you're playing."

Jayant is part of a multidisciplinary team of design, human-computer interface, human factors and prototyping experts that have been focused on finding natural, intuitive and immersive ways to use the RealSense camera technology.

RealSense camera technology allows people to control and interact with new PCs using hands gestures, not unlike the way actor Keanu Reeves did in the 1999 movie Matrix.

By taking a RealSense camera, the same technology built into many of today's laptops, 2 in 1 and all-in-one computers, and connecting it to a wearable computer strung with vibrating haptic actuators, Jayant's team has shown how computer vision can help sight-impaired people navigate the real world.

Now her team believes this technology could help give vision impaired and blind people a better feel and way to interact with augmented reality experiences, including games.

"Imagine if being able to take characteristics of our real-life physical environments, including biorhythms, and blending them into games and digital spaces," said Jayant.

New games like psychosocial thriller Nevermind and Laser Life are designed to allow gamers to augment games experiences using RealSense camera technology.

Using a RealSense-enabled computer, Nevermind captures a person's real-time heart rate. The game reacts to the player's emotional swings by intensifying or calming the experience. Laser Life lets people with RealSense equipped computers use their hands to gesture-control the game.

Jayant explained that there's no need for a camera to identify anything within a purely digital gaming experience, but one is needed if the game is designed to pull in elements from the real world.

As more developers allow their games to be augmented with data generated from computer vision technologies used by blind people to navigate their daily lives then blind and vision impaired people would have some familiarity inside gameplay and virtual landscapes, according to Jayant.

Earlier this year, at the International Consumer Electronics Show (CES), RealSense camera technology brought 3D vision to robots and allowed drones to fly autonomously through obstacles.

It's where Jayant's team first showed their working prototype of the RealSense Spatial Awareness Wearable, a vest fitted with a computing module that connects wirelessly to eight thumb-sized vibrating sensors: three across the chest, three across the torso and two near the ankles of each leg.

The wearable system is designed to send impulses to its wearer's body based on cues from real-world surroundings.

Darryl Adams, a technical project manager at Intel who was diagnosed with Retinitis Pigmentosa (RP) nearly 30 years ago, has been helping the RealSense research team test prototypes.

"This system is high resolution," said Adams, who said the Spatial Awareness Wearable helped him to get a better awareness of his surroundings, especially up close, where most social interactions take place.

When someone or something gets near him, he feels a vibration on his body that indicates something is approaching from his right or left.

"It's like sonar telling you something is in your way, but this one sees more details."

It was after CES when researchers realized that their prototype, which gave Adams a better sense of reality, might have the chance to help blind people enjoy better virtual or augmented reality games, according to Rajiv Mongia, director of the RealSense Interaction Design Group.

"We're using a sense of touch — in this case, vibrations on different parts of the body—as a way to give people a sense of sight," said Mongia, who believes this sense of sight in reality is something that could be used in games and augmented experiences.

"The sense of touch is so immediate to me," said Adams, who agrees that haptics and RealSense technology could improve some gaming and virtual reality experience for him.

"First-person shooter games have become so sophisticated with so many stats and tools that come into play. I can't see those things. Period. If I tried to play, I'd be shot every time."

He said that gaming seems like a natural place for the RealSense assistive technology to evolve, but it could be challenging to optimize haptic language so that it's interpreting things that you're missing visually.

A whole language — personal or universal — based on computer vision and haptics could be created using different parts of the body, according to Adams.

"Different patterns could be immediately interpreted as something that seeing people typically get through visual cues," he said.

With this physical-digital blending, Jayant said game developers could use the same feedback cues that would be used when interacting in real, physical settings.

"In real life, if you're walking down the street and there's an overhanging branch, then the system's top haptic motors worn by the person will go off. As you get closer to the tree branch, the vibrations get stronger and stronger, signaling to the wearer to duck or walk around that branch.

"During a game, if any person walks by, a visually impaired player could actually feel it as a wave going by them."

Jayant said that her team is not yet blending their real world navigation system with particular games, but her team sees great potential.

"This provides even more immersive and multimodal feedback for everyone," said Jayant.

As her team refines their existing prototype — bringing benefits of facial recognition and catalogs of real life objects, places and experiences to help blind people navigate the real world — Jayant will explore how computer vision can bring blind people strong sensibilities and quick understanding of the world inside games and virtual settings that someday might leverage the RealSense technology.

"This is technology that is still waiting to be realized, but the potential for the blind to join everyone else in virtual worlds is huge," said Jayant.

• 

• 

•