Mixed reality cookbook

July 14, 2010, ICT Results
Mixed reality cookbook

What we perceive in the world is highly influenced by what we are looking for. That is old news. Now European researchers have used this theory to create a convincing and engaging 'mixed reality', and they have put together a cookbook so others can do it, too. That is new news.

Reality is WYSIWYG: What You See Is What You Get. But what you see depends largely on what you are looking for.

In a famous experiment, a group of volunteers observed a video of two teams, one dressed in black and one in white, passing a ball between them. The volunteers had to count the number of times the ball was passed directly from one player in black to another player in black. They performed the task excellently.

What they failed to notice was the man in the gorilla suit who walked on screen and jumped up and down during the game. It proved that what you see is strongly influenced by what you are looking for.

Non-linear fuzziness

In ophthalmology, researchers have found the eye does not see everything you perceive; neural processing fills in parts of the scene by inferring from those bits that are observed. In quantum physics, researchers discovered that particles change behaviour depending on whether you are looking at them or not.

In field after field researchers have discovered that perception is not linear; it is fuzzy; and it can be strongly influenced by carefully choosing the right cues.

The cues do not necessarily require complex technology. The Wii, a very popular , abandoned the arms race of ever-more powerful processors and graphics cards and instead incorporated a simple . Now users' and reflexes drive the game, changing the pastime from a solitary, passive experience into an active, social one.

Those two additions, sociability and physicality, dramatically enhance the sense of experienced reality engendered by the game. This is very interesting. Up to now technologies, such as virtual and mixed reality, were thought by most to rely on more power, more technologically advanced interfaces, more animation and textures; but it now seems mixed reality is more powerfully and realistically evoked by combining perceptual dimensions with novel technologies in order to create a greater depth of experience.

“The greater the combination of senses engaged, the greater the chance of the user feeling immersed or present in the experience,” explains Rod McCall, a researcher at the Fraunhofer Institute and coordinator of the IPCity project.

In IPCity, a major EU-funded mixed reality project, researchers studied dozens of technologies to find those that dramatically enhance a user’s experience of a given task, all in an effort to increase citizens’ participation in civic life.

V-Ex and the city

Using virtual experiences (or V-Ex if you want) like this to bring citizens closer to the city, the project  embarked on what is probably the largest concerted effort, looking at the widest variety of mixed reality implementations, in recent times.

The project created applications for town planning, gaming, environmental awareness and storytelling. It enhanced engagement with the social, cultural and historical fabric of a city through location awareness and mapping, and it developed social storytelling rooted at locations within the streetscape.

Using a combination of easy-to-understand yet state-of-the-art technologies and location sensing, the researchers were able to create convincing cross-reality experiences by engaging multiple senses in parallel.From the lab to the real world

The project took perceptual and mixed reality research out of the lab and into the real world with a combination of large-scale field trials and longitudinal studies. As a result, the IPCity team has developed cookbook-like guidelines for creating mixed reality experiences.

Take Urban Renewal, an urban redesign application. Here, the researchers used a wide variety of media and interfaces to engage citizens in an exercise for redesigning an urban space.

IPCity’s Colour Table is a particularly innovative interface, using tokens to represent elements within a scene, such as buildings or other objects. An overhead camera projected the design table onto a wall, revealing changes as they developed from a bird’s eye view.

Another camera ‘interprets’ the tokens and projects virtual mock-ups onto a backdrop of the real site. Meanwhile on a screen, users can see how they have arranged the tokens, and on another they see how that would impact the real landscape.

The entire set-up, along with other tools, is part of a mobile tent that is transported to the actual location for the new building, so participants can visualise the real-world environment. The combination of these technologies, along with subtle audio streams, evokes a very convincing air of engagement in the task.

Token reality

“In the Urban Renewal showcase you have coloured tokens on the table and these represent buildings or another object in the space. So rather than having participants moving graphical objects on the screen you have them physically moving real objects on the table,” stresses McCall.

This physicality makes it easier for other people who are there to grasp what's going on - it starts a much richer discussion around the design through this physical relationship with it.

Another showcase by IPCity, CityWall in Helsinki, uses a very large multi-touch screen on a central street. “People can do whatever they want with content which is uploaded to flickr. But they appropriate the CityWall in hundreds of different ways. People play 'Pong' with the content, throwing it across the screen to one another, and it becomes [an] expressive space, far more perhaps than people showing pictures on their computer monitor,” McCall notes.

A sociable science

Social elements form a part of all the applications developed by IPCity, deliberately so. “We figured out pretty early on that a shared experience is much richer, that is why two people work together in our game, TimeWarp. It becomes a shared, new reality,” McCall explains.

Active, physical engagement, too, is important. McCall reveals that because it is impossible to cover a city entirely in virtual objects, the project developed activities that players need to complete, such as walking through a time portal or following a beer cart through the city. As non-players cannot see what the players are doing it often led to confused looks by passers-by in Cologne when it was demonstrated there.

In many respects, IPCity’s work represents the state of the art for mixed reality experiences and promises to offer a lot of food for thought to tourism, social gaming and mobile phone companies, among many others, including the performing arts. In fact IPCity’s work could be applied in some fashion to almost any area, to dramatically enhance the experience.

IPCity has applied the theory on in novel, compelling ways and its work will go on to enhance the theory further and lead to even more subtle and ingenious applications of V-Ex, in the city and elsewhere.

This is the first of a two-part special feature on IPCity.

Explore further: Real and virtual pendulums swing as one in mixed reality state

More information: IPCity project - www.ipcity.eu/

Related Stories

Pendulum Finds Virtual Soulmate

June 14, 2007

What's nerdier than creating an online avatar that fights dragons and raids strongholds? Creating a virtual pendulum that you can sync up to your real-life pendulum. Leave it to physicists to do just that, resulting in a ...

Pervasive games promise to spice up daily life

September 11, 2008

(PhysOrg.com) -- In the movie The Game, the character is hounded by villains and left for dead in Mexico in an intense version of an alternative reality game. Minus the Hollywood bravado, games that merge the virtual with ...

Augmented Reality: Science Fiction or Reality? (w/ Video)

July 7, 2009

(PhysOrg.com) -- Computer graphics have come a long way since the birth of Atari Games over 30 years ago. Today, computer graphics seem very real and some day researchers will pull graphics out of your television or computer ...

Augmented reality brings movie magic to city visits

April 14, 2010

(PhysOrg.com) -- From the Italy of Visconti to the Spain of Almodovar, many people's first encounter with a foreign country is in the cinema. Imagine if images from the silver screen could be conjured up anew when they do ...

Recommended for you

Cryptocurrency rivals snap at Bitcoin's heels

January 14, 2018

Bitcoin may be the most famous cryptocurrency but, despite a dizzying rise, it's not the most lucrative one and far from alone in a universe that counts 1,400 rivals, and counting.

Top takeaways from Consumers Electronics Show

January 13, 2018

The 2018 Consumer Electronics Show, which concluded Friday in Las Vegas, drew some 4,000 exhibitors from dozens of countries and more than 170,000 attendees, showcased some of the latest from the technology world.

Finnish firm detects new Intel security flaw

January 12, 2018

A new security flaw has been found in Intel hardware which could enable hackers to access corporate laptops remotely, Finnish cybersecurity specialist F-Secure said on Friday.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.