Augmented reality to help astronauts make sense of space

Sep 22, 2009
Frank de Winne's WEAR training session.

( -- Life aboard the International Space Station is hard work. Crewmembers have a multiplicity of complex tasks, potentially involving thousands of tools, components and other items. But ESA astronaut Frank De Winne has begun testing the prototype of an unusual helper designed to make astronaut life easier.

The ESA-designed Wearable (WEAR) is a wearable computer system that incorporates a head-mounted display over one eye to superimpose 3D graphics and data onto its wearer’s field of view.

Controlled by voice for hands-free operation, WEAR includes onboard location and object identification to show precise information about what they are looking at, as well as providing step-by-step instructions to guide them through difficult, lengthy procedures.

“At the moment, (ISS) crews still use paper instruction manuals for many operational and maintenance tasks,” explains Luis Arguello of ESA’s Modelling and Simulation Section, overseeing the WEAR project. “Obviously, it’s easier to perform a task while holding instructions in your hand. So we have developed a new type of user interface that is easier still, allowing astronauts to be guided precisely in their work without holding anything at all.”

From virtual to augmented reality

While the better-known concept of ‘’ (VR) concerns immersive artificial worlds, ‘augmented reality’ (AR) involves blending the real world with simulated elements. The technology has already been employed in niche areas such as flight simulations and surgical training, but AR’s range of uses has increased as available has grown.

“The WEAR concept started as research and development within ESA’s General Support Technology Programme, targeting commercial non-space applications such as architecture and maintenance as well as space activities,” explains Mr Arguello. “It’s the result of many years of VR/AR research. The system took a long time to conceive but was very fast to implement: once WEAR was accepted by ESA and NASA, we had to work fast to turn a design concept into flight-ready hardware."

Reflecting the pace of development, WEAR has been assembled from largely off-the-shelf components. Key hardware elements include a mobile computer connected to a headset with a head-mounted display, a pair of video cameras - for barcode reading and objection recognition - and an Inertial Measurement Unit (IMU).

The image-recognition system allows WEAR to know where its user is located, by checking his viewpoint against stored 3D information about the module, derived from a Columbus Computer Aided Design (CAD) model. The IMU works like the orientation tracking technology on an Apple iPhone, letting WEAR assess in which direction its wearer is moving and update his position and the image displayed accordingly.

Tools and documents on-board ISS.

On the software side, WEAR incorporates a 3D AR toolkit as well as speech recognition and synthesis technology, object recognition and tracking systems and commercial barcode reading technology. Reading the barcode allows the quick identification and retrieval of information of ISS items stored in the onboard Inventory Management System.

Technology demonstrator

The WEAR system, which was carried to the ISS on the Space Shuttle in August, is a technology demonstrator rather than a fully operational model, mainly due to schedule and budget constraints and some technical limitations in the computer hardware available on the ISS.

All equipment flown aboard the ISS must go through arduous safety and compatibility testing, and it would have taken too long to pass a faster ultramobile PC rather than use the IBM 31P Thinkpads already qualified and aboard the ISS. While state-of-the-art when first flown, these laptops are now more than five years old, though the good news is that more modern A61 Leonovo laptops are in the process of being introduced and should be available for future WEAR tests.

This demonstrator began its testing in ESA’s Columbus module on 12 September. An ISS Thinkpad has a battery life of only an hour with WEAR attached to it, so a task was chosen with that limitation in mind: opening up decking on Columbus to replace an internal filter. Frank De Winne was able to navigate by voice through each step of the procedure and visualise all the relevant information, diagrams and location information with WEAR, keeping his hands free for the maintenance work.

“I invited Frank in the early phases of the project to take part in reviews as a potential user in order to capture user requirements,” says Mr Arguello. “He became our main supporter and ended up presenting the idea to NASA.”

WEAR will be the third 3D visualisation system flown by ESA to ISS. Last year’s Automated Transfer Vehicle (ATV) delivered a pair of laptop-based training systems to ‘refresh’ astronauts in their knowledge of ATV systems.

This ESA project has Belgium-based Space Applications Services as its prime contractor, working with the Katholieke Universiteit Leuven as subcontractor in charge of the vision-based localisation software.

Meanwhile, on the ground, Space Applications is considering firefighting as a non-space application of WEAR. Within the space sector, it is being proposed to support operations inside ESA’s test facilities.

Provided by European Space Agency (news : web)

Explore further: SpaceX close to figuring out rocket failure during launch

Related Stories

Personal digital assistants in space

Jan 26, 2007

Can tiny and ubiquitous devices like Personal Digital Assistants (PDAs) be of use for space applications? The answer is a definite yes. Recent tests have demonstrated current and future uses for PDAs on board ...

Columbus installed in new home on ISS

Feb 11, 2008

The European Columbus laboratory has completed its voyage to the International Space Station. Columbus was officially attached to the right side of the Harmony module at 22:44 CET this evening.

Columbus, one year on orbit

Feb 11, 2009

Exactly one year ago today, the European Columbus laboratory arrived at its berth on the International Space Station, signalling the start of a new era for Europe in human spaceflight.

Atlantis readies for Columbus mission

Jul 13, 2007

More than 500 years ago, the voyages of Christopher Columbus to the New World and the discoveries he made caused quite a stir throughout Europe. Today, excitement is building on both sides of the Atlantic ...

The European Columbus space laboratory set to reach ISS

Dec 03, 2007

With NASA’s announcement today of the launch of Space Shuttle Atlantis on 6 December, ESA astronauts Hans Schlegel, from Germany, and Leopold Eyharts, from France, are set to carry ESA’s Columbus laboratory ...

Unique three-way partnership for ATV ground control

Feb 26, 2008

Shortly after its planned 8 March launch to the ISS, ESA's Automated Transfer Vehicle will make the first-ever laser-guided docking in space. The vehicle's fully automated docking provides unique challenges to the multi-agency ...

Recommended for you

What is the newest planet?

13 hours ago

With astronomers discovering new planets and other celestial objects all the time, you may be wondering what the newest planet to be discovered is. Well, that depends on your frame of reference. If we are ...

Catching Earth at aphelion

13 hours ago

Do you feel a little… distant today? The day after the 4th of July weekend brings with it the promise of barbecue leftovers and discount fireworks. It also sees our fair planet at aphelion, or its farthest ...

Opportunity's 7th Mars winter to include new study area

14 hours ago

Operators of NASA's Mars Exploration Rover Opportunity plan to drive the rover into a valley this month where Opportunity will be active through the long-lived rover's seventh Martian winter, examining outcrops ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.