A computer system allows a machine to recognize a person's emotional state

To detect the user's emotional state, the scientists focused on negative emotions that can make talking with an automatic system frustrating. Specifically, their work considered anger, boredom and doubt. To automatically detect these feelings, information regarding the tone of voice, the speed of speech, the duration of pauses, the energy of the voice signal and so on, up to a total of sixty different acoustic parameters, was used.

In addition, information regarding how the dialogue developed was used to adjust for the probability that the user was in one emotional state or another. For example, if the system did not correctly recognize what the interlocutor wanted to say several times, or if it asked the user to repeat information that s/he had already given, these factors could anger or bore the user when he was interacting with the system. Moreover, the authors of the study, which has been published in the Journal on Advances in Signal Processing, point out that it is important that the machine be able to predict how the rest of the dialogue is going to continue. "To that end, we have developed a statistical method that uses earlier dialogues to learn what actions the user is most likely to take at any given moment", the researchers highlight.

Once both emotion and intention have been detected, the scientists propose automatically adapting the dialogue to the situation the user is experiencing. For example, if he has doubts, more detailed help can be offered, whereas if s/he is bored, such an offer could be counterproductive. The authors defined the guidelines for obtaining this adaptation by carrying out an empirical evaluation with actual users; in this way they were able to demonstrate that an adaptable system works better in objective terms (for example, it produces shorter and more successful dialogues) and it was perceived as being more useful by the users.

This study was carried out by Professor David Grill Barres, of the Applied Artificial Intelligence Group of UC3M's Computer Science Department, together with Professors Zoraida Callejas Carrión and Ramón López-Cózar Delgado, of the Spoken and Multimodal Dialogue Group of the Computer Languages and Systems Department of the UGR. This achievement falls within the area of affective computation (computer systems that are capable of processing and/or responding to the user's emotions).

More information: Predicting user mental states in spoken dialogue systems, Z. Callejas, D. Grill, R. López-Cózar, EURASIP Journal on Advances in Signal Processing, 2011:6, pp. 1-23

Provided by Carlos III University of Madrid