A 'social' virtual assistant for migrants

Migrants have been very high on the EU political agenda for the past few years. But far from the political debate, there are cases where technological innovation can truly make a difference. The KRISTINA project has been developing such technologies with a focus on overcoming language barriers.

As great as it sounds on paper, freedom of movement in Europe can pose enormous challenges. Imagine you get a great job opportunity thousands of kilometres away from home, in a Member State you have never even visited. Sure, it's still Europe, but on a daily basis dealing with the administration or telling the doctor about a health problem in a language you don't speak can quickly become insurmountable.

Here, a true 'multilingual conversational agent' capable of understanding and providing the information you need would be a game changer. "The idea would be to develop such an agent with emotive, social and cultural competence. One capable of: automatic speech recognition (ASR) with linguistic analysis of the transcripts to understand the concern of the speaker; facial and gesture analysis; dialogue planning that doesn't rely on predefined scripts; understanding of emotive signals; multilingual language generation techniques; and featuring advanced virtual character design," says Leo Wanner, ICREA research professor at Pompeu Fabra University specialising in computational linguistics.

At first read, this list of features can seem technologically out of reach, but with some advances in state-of-the-art technology it could be right around the corner. Actually, work under the KRISTINA (Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities) project has already enabled Prof. Wanner and his team to create the first generation of this conversational agent and run it on tablets and laptops.

The assistant is very simple to use. Once initiated (which requires going through some settings concerning the language, the desired topic of conversation and the like), KRISTINA's virtual agent appears on screen, greets the person wishing to interact with it, and enquires about the possible concerns or questions he/she might have. "Based on the reaction of the person, a conversation develops during which the agent may provide background information on health issues of interest. The agent can provide advice with respect to health, social and other types of activities, offer to read the newspaper to those with visual impairment, etc.," Prof. Wanner explains. KRISTINA can interact with people in German, Polish, Turkish and Spanish.

Whilst not a Big Data project per se, KRISTINA needed to be 'trained' to accurately recognise the emotions of the individuals in the targeted groups, learn the way they speak and interact. This required a considerable amount of data, such as recordings of conversations between individuals of the targeted groups and experts who simulated the agent. The team annotated these recordings with the relevant meta information, and Prof. Wanner says that "a good share" of this resource will be made available to the research community.

Prof. Wanner admits that it's still too soon for the KRISTINA agent to be deployed as an "off-the-shelf" conversational agent that can assist migrants (or another relevant target group) in their everyday lives.

"Further development is needed. Project partners continue to work on the individual technologies – although, unfortunately, not as a consortium due to the lack of common funding. We are working in order to change this situation, and our objective is to expand the KRISTINA agent to enable it to serve as an assistant in other domains," says Prof. Wanner. According to him, KRISTINA should reach maturity in its second or third generation.