Expert help from a distance

Machines stretch from one end of the production hall to another, each of them an important part of the manufacturing process. When one of these complex pieces of machinery stops working, the on-site technicians grab their tools and the manual and try to fix it – but sometimes the only solution is to call the manufacturer for tips on how to get it working again. The problem is that giving advice over the telephone is never easy: Do they mean the screw on the right or the screw on the left? Well, that depends on which side of the machine you are standing on! Even putting things down in writing can only get you so far. Often the only choice left is to fly experts in, sometimes from the other side of the world. But until they arrive, the machine stays stubbornly offline, perhaps bringing the entire production process to a halt.

Soon experts will have an easier and quicker method of supporting the on-site technicians without having to set foot outside their office: the augmented reality system developed by researchers from the Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE in Wachtberg. This allows technicians to record the malfunctioning machine with a camera fixed to the back of their laptop monitor. The computer is mounted on a swivel arm so that the technicians can view the screen while carrying out repairs. An image processing program calculates the camera's position and direction of view and sends this information to the manufacturer over standard telecommunications networks. This enables the experts to view the machine on their monitor from the same perspective as the technicians. They can even use the software to write instructions on specific parts of the machine such as 'Remove this screw'. These instructions then pop up on the technicians' screen on exactly the same part of the machine. And if a technician walks around the machine with the laptop, the image moves accordingly – and the written information stays where it was intended to be, for example hovering over a specific screw. Once the technicians have carried out the experts' instructions, the pop-up information can be deleted by simply clicking on it. The system is based on a chat protocol, which means everyone involved can communicate either through the chat function or by telephone.

The researchers managed to minimize the quantity of data transmitted to allow the system to function over a cellphone network. That means there is no need for a broadband connection, so technicians can call on experts even from remote locations such as wind turbines in the middle of a field or machines in newly industrialized and developing countries. "We only transmit location data, not pictures," says Dr. Thomas Alexander, who heads up the research team at the FKIE. "At the moment we do that by attaching 2D barcodes to various parts of the machine. When the repair technician takes a picture of the malfunctioning machine, the software on the laptop reads those markings and links them to information in the database – for example the machine's identification code and the position and location of the barcodes that appear in the picture. And this is the only information that actually gets transmitted." Once the data reaches the manufacturer, another software program links it to the machine's CAD data to enable the experts to view the machine from the same angle as the on-site technician. The researchers have already developed a prototype of the system, and the next step is to carry out a study in which users will put the system through its paces. The results should help the scientists to optimize the system and tailor it more closely to user requirements.

Provided by Fraunhofer-Gesellschaft