Touch- and gesture-based input to support field work

Fujitsu Laboratories today announced that it has developed a wearable device in the form of a glove, equipped with a Near Field Communication (NFC) tag reader and that features gesture-based input for maintenance and other on-site operations.

Smartphones, tablets, and other smart devices have made work more efficient by digitizing the process of activity-logging, and attempts have been made to use them to support work outside of office environments. But in some work settings, such as those that require gloves to be worn or environments in which hands get dirty, taking out and using a conventional smart device can be difficult. Another hurdle is that users need to stop what they are doing in order to use their smart device.

With the Fujitsu Laboratories-developed wearable device, users can receive work instructions and other information, just by taking the natural action of touching an object. Task results can also be input using gestures so that operations can be carried out smoothly.

One example of a solution obtained by using the device together with a head-mounted display is when an operator touches a connector or control panel's NFC tag to obtain its work order, then making a simple gesture to register the task's results. This is expected to streamline operations and cut errors.

This glove device will be exhibited at Mobile World Congress 2014, running February 24-27 in Barcelona.

As mobile devices have gotten smaller, communications technology more advanced, and cloud environments more widespread, smart devices have come to be used in factory or building maintenance and other on-site operations. They can be used to provide information in a timely fashion while performing a task, and therefore promise to make work more efficient, with fewer errors or oversights, even when the operator is not an expert.

Existing field-support technologies require the operator to manipulate a smart device to get information. Some work settings, however, are not amenable to the use of such devices as they involve the wearing of gloves or working with dirty hands, making it difficult to retrieve and use a conventional smart device. Furthermore, looking at the smart device to perform an operation interrupts the task at hand.

In order to provide useful information suited to a task that the operator should pay attention to, without requiring the use of a terminal, Fujitsu Laboratories has developed a glove device that combines an NFC tag reader with gesture-input functionality (Figure 1).

Using this device lets the operator simply touch the NFC tag of the object being worked on to display relevant information (Figure 2). A gyro sensor and accelerometer in the wrist enable the device to recognize gestures.

Key features of the technology are as follows:

1. Electric-control technology in a wearable device (Figure 3)

Wearable devices need to be comfortable without placing a burden on the user's body, and for that reason, they cannot have bulky batteries. Accordingly, low-power operation is a prerequisite. For the glove device, by equipping the operator's finger with a touch sensor and activating the NFC tag reader only for the moment the object is touched, power consumption is able to be kept low.

This gives the glove device nine hours of runtime, compared to three hours for a device that lacks this kind of power control for operating time. This longer battery life is sufficient for an entire workday.

The steps involved in the power-control operation are as follows:

1. Contact sensor detects touch of object
2. NFC tag reader in the glove device is activated
3. Tag ID data attached to object is read
4. Tag data is transmitted to a smart device
5. NFC tag reader goes into standby mode

2. Reliable gesture-recognition technology works with a variety of tasks and postures (Figure 4)

Fujitsu Laboratories has developed gesture-recognition technology that functions reliably in multiple tasks and postures without being awkward. Gesture-recognition accuracy is largely dependent on the ability to distinguish between gestures and normal arm movements during the course of work as well as variations in gestures between individuals. This recognition technology takes note when the wrist is in a dorsiflexed position, where the hand rotates back, a movement that is not common in everyday tasks, and activates on that position, so that it can distinguish between normal arm movements and gestures. It also defines gestures based on the shoulder as the center of a coordinate system. This accommodates variations between individuals and allows gesture-based inputs in a variety of positions without awkwardness.

In-house testing showed that six patterns (up, down, left, right, rotate left, rotate right) could be recognized with 98% recognition accuracy. These gestures could be used to flip through pages of a manual being shown on a head-mounted display with left and right motions, or to scroll with up and down motions. In a scenario where it is used to input work status, a right rotation could indicate a normal state, and a left rotation a problem.

This glove device could be used to develop a solution that allows for an operator to tap on the object being worked on to display work orders, and to input task outcomes with simple gestures, increasing efficiency in the field and cutting down on errors and oversights.

Fujitsu Laboratories plans to perform verification testing on the technology with the aim of bringing a product to market in fiscal 2015. In addition, through the technologies it develops, Fujitsu Laboratories seeks to accelerate the advent of its vision of a Human Centric Intelligent Society, where the power of ICT generates a wealth of creative innovations.