Augmented reality to help astronauts make sense of space

September 22, 2009
Frank de Winne's WEAR training session.

( -- Life aboard the International Space Station is hard work. Crewmembers have a multiplicity of complex tasks, potentially involving thousands of tools, components and other items. But ESA astronaut Frank De Winne has begun testing the prototype of an unusual helper designed to make astronaut life easier.

The ESA-designed Wearable (WEAR) is a wearable computer system that incorporates a head-mounted display over one eye to superimpose 3D graphics and data onto its wearer’s field of view.

Controlled by voice for hands-free operation, WEAR includes onboard location and object identification to show precise information about what they are looking at, as well as providing step-by-step instructions to guide them through difficult, lengthy procedures.

“At the moment, (ISS) crews still use paper instruction manuals for many operational and maintenance tasks,” explains Luis Arguello of ESA’s Modelling and Simulation Section, overseeing the WEAR project. “Obviously, it’s easier to perform a task while holding instructions in your hand. So we have developed a new type of user interface that is easier still, allowing astronauts to be guided precisely in their work without holding anything at all.”

From virtual to augmented reality

While the better-known concept of ‘’ (VR) concerns immersive artificial worlds, ‘augmented reality’ (AR) involves blending the real world with simulated elements. The technology has already been employed in niche areas such as flight simulations and surgical training, but AR’s range of uses has increased as available has grown.

“The WEAR concept started as research and development within ESA’s General Support Technology Programme, targeting commercial non-space applications such as architecture and maintenance as well as space activities,” explains Mr Arguello. “It’s the result of many years of VR/AR research. The system took a long time to conceive but was very fast to implement: once WEAR was accepted by ESA and NASA, we had to work fast to turn a design concept into flight-ready hardware."

Reflecting the pace of development, WEAR has been assembled from largely off-the-shelf components. Key hardware elements include a mobile computer connected to a headset with a head-mounted display, a pair of video cameras - for barcode reading and objection recognition - and an Inertial Measurement Unit (IMU).

The image-recognition system allows WEAR to know where its user is located, by checking his viewpoint against stored 3D information about the module, derived from a Columbus Computer Aided Design (CAD) model. The IMU works like the orientation tracking technology on an Apple iPhone, letting WEAR assess in which direction its wearer is moving and update his position and the image displayed accordingly.

Tools and documents on-board ISS.

On the software side, WEAR incorporates a 3D AR toolkit as well as speech recognition and synthesis technology, object recognition and tracking systems and commercial barcode reading technology. Reading the barcode allows the quick identification and retrieval of information of ISS items stored in the onboard Inventory Management System.

Technology demonstrator

The WEAR system, which was carried to the ISS on the Space Shuttle in August, is a technology demonstrator rather than a fully operational model, mainly due to schedule and budget constraints and some technical limitations in the computer hardware available on the ISS.

All equipment flown aboard the ISS must go through arduous safety and compatibility testing, and it would have taken too long to pass a faster ultramobile PC rather than use the IBM 31P Thinkpads already qualified and aboard the ISS. While state-of-the-art when first flown, these laptops are now more than five years old, though the good news is that more modern A61 Leonovo laptops are in the process of being introduced and should be available for future WEAR tests.

This demonstrator began its testing in ESA’s Columbus module on 12 September. An ISS Thinkpad has a battery life of only an hour with WEAR attached to it, so a task was chosen with that limitation in mind: opening up decking on Columbus to replace an internal filter. Frank De Winne was able to navigate by voice through each step of the procedure and visualise all the relevant information, diagrams and location information with WEAR, keeping his hands free for the maintenance work.

“I invited Frank in the early phases of the project to take part in reviews as a potential user in order to capture user requirements,” says Mr Arguello. “He became our main supporter and ended up presenting the idea to NASA.”

WEAR will be the third 3D visualisation system flown by ESA to ISS. Last year’s Automated Transfer Vehicle (ATV) delivered a pair of laptop-based training systems to ‘refresh’ astronauts in their knowledge of ATV systems.

This ESA project has Belgium-based Space Applications Services as its prime contractor, working with the Katholieke Universiteit Leuven as subcontractor in charge of the vision-based localisation software.

Meanwhile, on the ground, Space Applications is considering firefighting as a non-space application of WEAR. Within the space sector, it is being proposed to support operations inside ESA’s test facilities.

Provided by European Space Agency (news : web)

Explore further: Personal digital assistants in space

Related Stories

Personal digital assistants in space

January 26, 2007

Can tiny and ubiquitous devices like Personal Digital Assistants (PDAs) be of use for space applications? The answer is a definite yes. Recent tests have demonstrated current and future uses for PDAs on board the International ...

Columbus installed in new home on ISS

February 11, 2008

The European Columbus laboratory has completed its voyage to the International Space Station. Columbus was officially attached to the right side of the Harmony module at 22:44 CET this evening.

Columbus, one year on orbit

February 11, 2009

Exactly one year ago today, the European Columbus laboratory arrived at its berth on the International Space Station, signalling the start of a new era for Europe in human spaceflight.

Atlantis readies for Columbus mission

July 13, 2007

More than 500 years ago, the voyages of Christopher Columbus to the New World and the discoveries he made caused quite a stir throughout Europe. Today, excitement is building on both sides of the Atlantic as the long-awaited ...

The European Columbus space laboratory set to reach ISS

December 3, 2007

With NASA’s announcement today of the launch of Space Shuttle Atlantis on 6 December, ESA astronauts Hans Schlegel, from Germany, and Leopold Eyharts, from France, are set to carry ESA’s Columbus laboratory to the International ...

Unique three-way partnership for ATV ground control

February 26, 2008

Shortly after its planned 8 March launch to the ISS, ESA's Automated Transfer Vehicle will make the first-ever laser-guided docking in space. The vehicle's fully automated docking provides unique challenges to the multi-agency ...

Recommended for you

Hubble catches a transformation in the Virgo constellation

December 9, 2016

The constellation of Virgo (The Virgin) is especially rich in galaxies, due in part to the presence of a massive and gravitationally-bound collection of over 1300 galaxies called the Virgo Cluster. One particular member of ...

Khatyrka meteorite found to have third quasicrystal

December 9, 2016

(—A small team of researchers from the U.S. and Italy has found evidence of a naturally formed quasicrystal in a sample obtained from the Khatyrka meteorite. In their paper published in the journal Scientific Reports, ...

Scientists sweep stodgy stature from Saturn's C ring

December 9, 2016

As a cosmic dust magnet, Saturn's C ring gives away its youth. Once thought formed in an older, primordial era, the ring may be but a mere babe – less than 100 million years old, according to Cornell-led astronomers in ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.