Jazz-playing robot will provide insight into how computers communicate with humans

October 23, 2015 by Jodi Heckel, University of Illinois at Urbana-Champaign
Ben Grosser stands in front of his work that is part of the School of Art and Design Faculty Exhibition at Krannert Art Museum. Grosser’s current project involves designing a computer system that will communicate with humans through jazz improvisation. The project aims to provide insight into human-computer interaction. Credit: L. Brian Stauffer

When jazz musicians improvise alongside each other, they interact in ways that are as much about feeling the music as about thinking. But suppose one of the musicians is a robot. Can it improvise a jazz solo in response to what its human partner is playing?

A University of Illinois researcher is designing a robot – actually a computer system – that will communicate with humans through jazz improvisation and provide insight into artificial intelligence and human-computer interaction.

The project – called MUSICA (MUSical Improvising Collaborative Agent) – is part of the Defense Advanced Research Projects Agency's new Communicating with Computers program, which aims to "enable symmetric communication between people and computers," where computers are collaborators just like people are. The jazz robot is an unconventional approach to the problem.

"It's approaching this communication from a different perspective than linguistically, and maybe we can learn something new about computers and humans," said Ben Grosser, a professor of new media in the School of Art and Design. Grosser is also affiliated with the National Center for Supercomputing Applications, where he is helping develop a new initiative in critical technology studies.

"We proposed that music improvisation is a novel space for investigating communication because it's not just cognitive interaction," Grosser said of the MUSICA project. "It's also an embodied experience. You feel the music as much as hear it and think about it. You react."

To build a software system that might be able to improvise music, Grosser and his team will create a knowledge database of canonical jazz solos. Their approach is to computationally analyze these solos using "image schema," a way that people understand their world using spatial concepts. For example, a musician might play "inside," meaning their notes fit within what the song's chord changes suggest, or "outside," meaning they're pushing the boundaries of what is appropriate.

"It's a way of understanding what do and analyzing what happens between them," said Grosser, who has a long history of performance and interest in jazz, has an MFA in new media and a master's degree in music composition, and worked in the School of Music's Computer Music Project during graduate school.

He'll then add a performance system that will analyze what a human performer is playing in real time, including the beat, pitch, harmony and rhythm, and consult what it has learned about jazz solos to communicate musically in response. The system should "understand not just what should come next but when a response should happen in time," according to the project proposal. Grosser hopes to have a system that will be able to perform a sophisticated "call and answer" musical response in a year.

"The ultimate goal is that a human performer should perceive what it hears from the system as musical communication," he said. "I don't expect our jazz robot to be a Miles Davis. Maybe it can be a high schooler, if we can really nail it."

The project builds on previous work by Grosser, including an interactive robotic painting machine. The machine creates oil paintings while considering what it "hears" as input into the painting process.

Grosser said he is interested in not just , but also the cultural, social and political effects of software, and the question of whether a computer system can truly make art.

The jazz robot project "is valuable for evaluating computational systems, but also in how we think about what art is. I like the questions it poses," he said. "We're increasingly interacting with intelligent systems. How do the designs of these systems change our interactions with them and with other humans?"

Working with Grosser on the five-year, $2.4 million project are Kelland Thomas and Clay Morrison, professors in the University of Arizona's School of Information; and Colin Dawson, a math professor at Oberlin College.

Explore further: Study of jazz players shows common brain circuitry processes music and language

Related Stories

Astronomer creates music using star oscillations

August 13, 2015

(Phys.org)—Astronomer Burak Ulaş, with the Izmir Turk College Planetarium in Turkey has taken his work into a musical dimension, using star oscillations as a source for a musical composition. He has uploaded a paper describing ...

ASA: silence, jazz can reduce heart rate after surgery

October 14, 2014

(HealthDay)—Silence and listening to jazz music can reduce heart rate after surgery, according to a study presented at the annual meeting of the American Society of Anesthesiologists, held from Oct. 11 to 15 in New Orleans.

Recommended for you

Researchers engineer a tougher fiber

February 22, 2019

North Carolina State University researchers have developed a fiber that combines the elasticity of rubber with the strength of a metal, resulting in a tougher material that could be incorporated into soft robotics, packaging ...

A quantum magnet with a topological twist

February 22, 2019

Taking their name from an intricate Japanese basket pattern, kagome magnets are thought to have electronic properties that could be valuable for future quantum devices and applications. Theories predict that some electrons ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.