How eye tracking gives players a new experience in video games

March 29, 2016 by Eduardo Velloso, University Of Melbourne, The Conversation
The Division video games uses eye tracking technology to help target the enemy. Credit: Tobii EyeX/YouTube screengrab

Tracking people's eye movements is a concept that for a long time has captured people's imagination. More often than not, the technology has been depicted as part of rather dystopian futures: in the movie Minority Report, police could track your every move through your eyes; in Fifteen Million Merits, part of the television series Black Mirror, adverts would pause when you looked away, forcing you to pay attention.

The reality of eye tracking, however, can be a lot more fun than that. In particular, our research group has been interested in the applications of for gaming.

From the lab to the home

Eye trackers used to cost tens of thousands of dollars and were precious pieces of equipment at psychology labs. While expensive high-speed trackers still have their place in eye movement research, eye trackers for gaming purposes can now be bought for just a few hundred dollars.

Examples include the Tobii EyeX, the Steelseries Sentry and the EyeTribe. We are also starting to see laptopsthat ship with built-in eye trackers.

But these devices still lack a killer app, a compelling use case that provides a compelling reason for people to adopt the technology.

Gaming might very well suit this role.

An eye tracker is made of little more than a camera and a few LEDs to be attached to the bottom of a screen. The difference to a regular webcam is that they operate in the infra-red spectrum so that the variation in people's eye and skin colour does not interfere in the tracking.

The video will load shortly

Sophisticated algorithms then convert the image from the camera into a pair of coordinates that represent where you are looking at on the screen: the gaze point. Though deceptively simple, these coordinates offer a wealth of opportunities for developers to incorporate into their games.

What to do with the gaze point?

The simplest mechanic that can be built for the eyes is to use them as a cursor, similarly to how we use the mouse.

Variations of this include using them to select and place chess pieces, to move the weapon cross-hair in a first-person shooter game, and to replace the mouse enabling disabled users to play.

The video will load shortly

Gaze data can also be combined with the keyboard and mouse to create more immersive experiences. In Assassin's Creed Rogue, when the player looks at a point in the environment the character faces that direction, allowing for a more natural exploration of the environment.

Tom Clancy's The Division incorporates several gaze mechanics, including aiming, throwing grenades, looking for cover, and many more.

The advantage of using gaze is that developers can still reach players who do not own an eye tracker yet, but offer an extra experience for those who do.

The video will load shortly

The future of eye tracking in games

One of the coolest aspects of gaze as an input method is that it goes well beyond issuing explicit commands to games.

First, the eyes play an important role in social interactions between people and this can be incorporated into games. For example, characters can get upset if you look away when they are speaking or maybe engage in a conversation when you first notice them.

Second, the eyes reveal a lot about our cognitive processes, which can be used by the game's artificial intelligence to adapt the gameplay. For example, researchers have explored how to use players' gaze to infer their strategies in Hex, to predict their actions in Super Mario Bros and to modify the narrative in interactive storytelling. This gives games an almost mind reading ability that can wow players (or possibly freak them out).

The video will load shortly

Third, eye trackers can provide professional and hardcore players with a powerful analytics tool for them to evaluate and improve their performance. For example, players can record their throughout the game session and later watch the video to check whether they were paying attention to the right areas at the right times. A heatmap of gaze points can also give players a snapshot of their overall visual attention and highlight areas that require more focus.

A game-changer or just a gimmick?

Eye tracking has been around for a long time, but it is yet to reach the wide consumer market.

Recent examples, such as Pizza Hut's gaze-enabled menu, suggest that there certainly are interesting applications for the technology, but they still have to prove their value beyond gimmicky demos.

The Nintendo Wii Controller and the Microsoft Kinect have shown that gaming can be a powerful platform to demonstrate the capabilities of unusual input devices.

We will keep our eyes open to see if the same will happen for gaze.

Explore further: Eye-tracking software opens online worlds to people with disabilities (w/ Video)

Related Stories

Infants with blind parents pay less attention to eyes

November 19, 2015

For parents of young children, there are few milestones more memorable than that first word. But people communicate an awful lot to each other without ever saying anything at all. That raises an intriguing question: how do ...

Recommended for you

Apple issues update after cyber weapon captured

August 26, 2016

Apple iPhone owners on Friday were urged to install a quickly released security update after a sophisticated attack on an Emirati dissident exposed vulnerabilities targeted by cyber arms dealers.

Auto, aerospace industries warm to 3D printing

August 25, 2016

New 3D printing technology unveiled this week sharply increases the size of objects that can be produced, offering new possibilities to remake manufacturing in the auto, aerospace and other major industries.

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

DonGateley
not rated yet Mar 29, 2016
Where this will be a game changer is in foveated rendering where only the part of the image where they eye is gazing is rendered in full resolution with considerably reduced resolution at larger angles. Much higher resolution displays are needed before people can begin to approach the cinematic resolution they get in other ways such as just watching high def television. But if the entire field of view is rendered at such resolution computation and bandwidth becomes rapidly prohibitive. This is probably the only way to give people the viewing experience in a head mounted display that will drive the technology mainstream.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.