Medical monitor with eyes and ears

In intensive care Units (ICU), every second counts. In emergencies, doctors and nurses need to make the right decisions quickly. Fraunhofer researchers have developed a smart "proxemic monitor" which has optimized the processes in the most sensitive area of a hospital and clearly shows the data of the connected medical devices while avoiding false alarms. The screen can be controlled from a distance, without contact, by means of gestures and voice commands, thereby reducing the risk of transmitting pathogens.

Everything had just been quiet in the control room of the intensive care unit. Suddenly, though, there is excitement: the alarms on several medical devices in different rooms are sounding. Monitors are flashing and beeping. A drop in blood pressure, cardiac arrhythmia – the computer monitors show exactly what has to be done and where. Doctors and nurses rush to the rooms of the patients. Medical equipment there provides detailed information on the status of the critically ill patients. "It's not easy to keep track of everything here during hectic situations", says Paul Chojecki, scientist from the "Vision & Imaging Technology" Department at the Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institut, HHI, in Berlin.

Controlled by contact-free gestures

Together with his team as part of the joint project "control room", the scientist has developed an intelligent monitor. It quickly shows doctors and nurses the most important information about the vital signs of their intensive care patients. The screen has interfaces to the medical equipment in the room, as well as with the information systems in the hospital, and can be controlled by contact-free gestures and voice commands. The graphics of its display adapt to the distance from which it is viewed. "From the door, the doctor sees the data in a correspondingly large size. As he gets closer, the screen displays detailed information", Chojecki explains. A 3D camera provides the necessary data. The user interface is programmed based upon the web, so it is also suitable for mobile devices, such as tablets. The scientists are presenting a demonstration of the "proxemic monitor" at CeBIT 2016 (Hall 6, Stand B36). A practical test in cooperation with Uniklinik RWTH Aachen is planned for later this year.　

The system evaluates the data of the medical devices on the basis of the smart alarm design of the project partner, the Medical Engineering Department of Aachen University Hospital. This prevents false alarms. "Intensive care doctors have told us that this is a big problem. Current devices stick stubbornly to determined limits, without including all of the relevant factors that are necessary in order to comprehensively assess the risk situation. In addition, the acoustic signals produce a very loud noise, which is not beneficial to the health of either the staff or the patient", Chojecki says.

Fewer pathogens transmitted

Another advantage of the gesture control is that the doctor or health care provider does not have to touch the devices directly. "The transfer of pathogens in hospitals, particularly in intensive care units, is still a problem. The obligatory hand hygiene is sometimes forgotten, and viruses as well as bacteria are carried from room to room", Chojecki describes. The smart screen can be programmed at the bedside, as well as used in the control room (with a different configuration). Chojecki adds: "That's an important distinction, since it is not legally permitted for all of the instrument functions in the patients' rooms to be accessed from the control room. It has to be ensured that the doctor actually examines the patient before changing the treatment".

Three different cameras and a microphone scan the area in front of the monitor. Using the video data, the built-in software of the HHI analyzes whether there are people in the room, how far away they are from the screen, and what movements they are making. Depending on the distance, the display and functionality of the monitor changes. "Our monitor distinguishes between near, medium, and further distance. The cameras cover a maximum distance of four meters", Chojecki explains. From the medium distance, the cursor can be controlled with arm movements, and commands or short reports can be input by voice. With pre-programmed gestures, for example, a video call can be started, in order to have discussions with other physicians within or outside of the hospital. "We have given the monitor eyes and ears so as to allow for multi-modal interaction between the user and the system. Our software records distances and movements of the user in a contactless manner, interprets them, and converts them into commands for operating systems or machines", Chojecki explains.

3 questions for Paul Chojecki

Scientist, Vision & Imaging Technology Department, Fraunhofer HHI, Berlin

1. What are the next steps in this project?

We want to display more information, such as context-sensitive workflows, on the proxemic monitor so as to aid in the decision-making process. Another goal to reach by 2017: Integration of camera-based detection of users, as well as of their roles and authorizations. As a result, information and control elements can be provided more easily to individuals who are authorized and denied to those who are not.

2. What are the current trends regarding the topic of "Vision & Imaging Technology"?

Very evidently applications for augmented and virtual reality (VR): with professional 360° recording systems, the real world can be represented ever more accurately in VR glasses. With contact-free interaction via gestures, the user can move more easily in virtual worlds. Moreover, it is possible to reconstruct people in 3D and involve them in interactive TV formats or VR worlds.

3. What other topics are you working on at the moment?

We are describing and standardizing gestures. There are more and more systems and solutions with contact-free interaction. Concerning this area, it is important that more attention be paid in the future to ensuring that standards are adhered to. As a result, users will not have to re-learn the operating concept every time they want to control a system by gestures. For this purpose, we are currently developing a gesture description system and cooperating with other partners on the standard "DIN SPEC 91333 – Contact-Free Gesture Control for Human-System Interaction."