Depth-sensing technology enhances games

Three new games use Intel RealSense camera to pull players into intuitive, immersive and surreal experiences.

For anyone who has tried to get down to a catchy pop song while holding a controller during a round of Just Dance, or missed a clutch tennis shot because the Wii didn't sense the swing, hands-free depth-sensing technology is a saving grace.

When players can control a game using gestures and a computer that "sees" like a human, the options become a whole lot more interesting.

Enter Intel's RealSense camera, which allows users to do everything from change their background during a video chat to scan 3D objects. The 3D depth-sensing camera technology has found its way into an array of tablets, laptops and all-in-one PCs, and has ignited the imaginations of game creators and players alike.

"We are interested in using gestural control to create a feeling of transformation and a real sense of tactile wonder in the space between you and your computer screen," said Robin Hunicke of Funomena when describing the company's upcoming origami-inspired game that will use Intel RealSense technology.

"We're always interested in pushing the boundaries of what games can express," she said. "But even as we're building something that challenges your assumptions, we're actively embracing your input as a player."

Certainly, there are plenty of cool, technophilic features to geek out over. Players can control a game with their whole bodies or a tiny flick of their pinky fingers. Users can play a hologram keyboard or scan a beloved pet cat into Minecraft and make him explode—if he will sit still long enough.

As the new crop of depth camera-supported games prove, RealSense brings new play experiences that are fun, inventive and even therapeutic (well, except for the cat).

Laserlife

Taking its cues from electronic producers and the club scene, the game Laserlife is all about moving and grooving to thumping music.

It works as a showpiece for RealSense's gesture controls, letting players control an in-game character with their bodies, even though users are playing as an alien intelligence with laser beams instead of human arms.

"We primarily do depth detection—stuff you cannot do with a regular camera," said Alex Neuse, the co-founder of Choice Provisions.

Because RealSense has an internal infrared camera and laser projector, it can detect the distance between objects. It also knows if an object is in the background or foreground.

The results are gestures that make intuitive sense. Instead of inflexibly absorbing the memory molecules as they pass into a player's hands, the player actually has to catch them.

"It's almost like we're designing the functionality of the controller, and the controller is your body," Neuse explained.

As such, the team found itself imagining what would make bodies good controllers. They experimented with what felt natural while sitting upright in front of a computer.

The end result is that players snake their hands like they're hanging their arm out the window of a moving car and hand-surfing the wind, while also shooting lightning from their fingertips, Emperor Palpatine-style.

Neuse, who previously had a big role in the Bit.Trip games, found designing a game around natural human interface to be liberating.

"As much as I love this," he said, holding up an Xbox gamepad, "It is very restrictive in a sense."

The problem, he explained, is that designers are always thinking about how to make a game that users can play with the traditional gamepad. Thus gamers tend to get more of the same.

Games based on features like sense of depth, however, are a relatively unexplored frontier, which Neuse said fosters creativity and innovative mechanics. Laserlife is one of the first exploratory steps.

Nevermind

A 2015 award nominee at the IndieCade game festival, the spooky Nevermind relies on RealSense technology's biometric capabilities to measure a player's pulse. It then uses that data to ramp up the chills, scares and surrealism as necessary.

"It feels almost like magic," said Erin Reynolds, the game's creative director.

The ultimate adversary in Nevermind is not some final boss, but the player herself. A jittery player must learn to confront anxiety and slow her racing heart rate in order to progress.

More than a clever idea, biofeedback technology could have a big impact on games at large.

Biometric-reading cameras offer feedback loops that enable developers to design games around body data. One logical use for this is to adjust a game's difficulty based on how stressed out or comfortable the player is.

"You don't have to make a game that is one size fits all," she said. "The way it works is that the camera is sensitive enough to read tiny fluctuations in the skin tone."

Every time the heart beats, blood gets pumped through the vessels in a person's face, making her complexion ever so slightly rosier. Though this is invisible to the naked eye, the camera can read it, which indicates just how precise the technology is.

Beyond games, camera-based biometrics could bring an emotional component to computing more broadly. Reynolds believes that in the future our smartphones and laptops will use biometric data to "know what is going on subconsciously with the user."

For an example, Reynolds foresees computers that know when users are upset and will try to manage that agitation before they can hit send on an angry text or email.

"There is a lot that can be done to personalize the experience to the user and how they feel at any given moment," she said.

"It feels like we are living in the future."

Intel Block Maker for Minecraft (tentative title)

After weeks spent building a manor from Minecraft blocks, it is only natural that one would want to fill it with stuff. An upcoming app makes use of RealSense's 3D scanning technology to do exactly that.

Since RealSense sees the world in 3D space instead of as a flat image, the process is as simple as holding a favorite action figure or toy mech in front of the camera and slowly rotating it.

Voilà! The scanned object is transferred into the world of the game in voxel-based glory.

"It's not meant to be a hardcore modding tool," said Mark Day, the app's chief developer. "It's something that a 10-year-old could use."

Being that this is Minecraft, though, there is one minor, destructive catch. Once inside the game, the object takes on the properties of dynamite, so players must be very careful not to detonate that bonsai while decorating the virtual house.

"We're kind of using the tech in a way nobody intended," Day said, noting that he sees no reason to stop at blocky objects.

A number of facial scanning and tracking applications like FaceShift outfit virtual avatars with facsimiles of the viewer's face. But Day believes that as 3D cameras replace the standard cameras and webcams on PCs and devices, the player will have the option to scan their entire body into games.

Just imagine how epic all those selfies in Grand Theft Auto will be then.

• 

•