Researchers develop virtual composer assistant

Experts from the Institute of Cyber Intelligence Systems at the National Research Nuclear University MEPhI (Russia) are currently developing a virtual composer assistant that will be able to analyze the emotional state of music composers and follow their logic. Despite the high level of development of music theory, the process of creating music is still hard to formally algorithmize, since it is inseparably linked with the emotional experiences of the composer. This aspect of the creative process is of particular interest to experts in machine intelligence.

Scientists can already create synergy between humans and software in a way that allows the computer running the software to gather data on the emotional state and the intentions of the creator, and thereby become a continuation of the human mind. The virtual composer assistant is software that can create human-level music of high aesthetic quality on its own. It is a creative smart assistant that can add its own notes, chords and combinations to a melody written by the composer.

In order to achieve this goal, researchers had to identify musical sounds, chords and their combinations, create semantic maps based on this analysis, and connect them to models of humans' emotional perception of music. The new software is fundamentally different from other approaches, since the virtual assistant can maintain emotional contact with the composer, said Professor Alexei Samsonovich from the MEPhI Institute of Cyber Intelligence Systems.

"Most such programs represent neural networks taught to write music or make art," he said. "But our idea was to identify the emotional state of the composer and follow its logic."

This method is based on using semantic maps instead of neural networks, Samsonovich said. In semantic maps, the emotional state of one's mind is represented by a point in the affective space. In this space, every possible action has its own coordinates, and there is a certain rule that allows researchers to connect various coordinates with the others by following the logic of emotional perception. The laws of musical harmony and rhythm impose certain limitations on this process.

"These limitations, however, cannot define everything; they leave some freedom for creativity," Samsonovich explained. "This is the freedom that our software uses, while being guided by the semantic map and the rules of choosing the coordinates on it, and while taking into account the evolution of the emotional state. I would like to point out that we are not talking about machine learning, as the model is based on the principles of psychology (basic psychological aspects of behavior). In this sense, unlike neural networks, models like this can explain the essence of the phenomenon of creativity."

The mutual understanding between the composer and the virtual assistant can be achieved through organizing and storing various types of emotional reactions to musical elements—chords, their sequence and intervals—in the software environment.

In order to ensure storage and use of the information about the most successful and fitting combinations of sounds, the virtual assistant uses a semantic map, which allows the virtual composer not only to build on a melody set by the user according to his taste, but also monitor his mood, suggesting different versions and creating the proper accompaniment.

In the course of the method's development, researchers used a separate application in order to test how participants evaluate various combinations of chords according to three measures, each judged on a scale from one to 10: pleasant-unpleasant, cheerful-sad and exciting-boring. This evaluation system allowed researchers to create three-dimensional semantic maps.

"We assumed that one's choice of behavior based on social emotions, feelings and relations can be described by relatively simple dynamic equations," Samsonovich said. "The simple model of behavior based on the semantic map and two equations is indistinguishable from the human's behavior in the simplest of paradigms of social interaction in a virtual environment."

Samsonovich also added that the most promising approach in this field of research is based on cognitive architectures that are created using psychology and neuroscience data—the so-called biologically inspired cognitive architectures (BICA).

The researchers believe that the new development can be scaled up and used not only in music, but also in many other areas of digital creativity, as well as in human-machine systems. Creating and implementing such innovations is an important step in the study of the socio-emotional aspect of intelligence, which is responsible for the recognition and expression of emotions, intentions, motivations and desires. The key results of this research were published in the Procedia Computer Science journal.