Robot uses deep learning and big data to write and play its own music

June 14, 2017 by Jason Maderer

A marimba-playing robot with four arms and eight sticks is writing and playing its own compositions in a lab at the Georgia Institute of Technology. The pieces are generated using artificial intelligence and deep learning.

Researchers fed the nearly 5,000 complete songs—from Beethoven to the Beatles to Lady Gaga to Miles Davis—and more than 2 million motifs, riffs and licks of music. Aside from giving the machine a seed, or the first four measures to use as a starting point, no humans are involved in either the composition or the performance of the music.

Ph.D. student Mason Bretan is the man behind the machine. He's worked with Shimon for seven years, enabling it to "listen" to music played by humans and improvise over pre-composed chord progressions. Now Shimon is a solo composer for the first time, generating the melody and harmonic structure on its own.

"Once Shimon learns the four measures we provide, it creates its own sequence of concepts and composes its own piece," said Bretan, who will receive his doctorate in music technology this summer at Georgia Tech. "Shimon's compositions represent how music sounds and looks when a robot uses to learn everything it knows about music from millions of human-made segments."

Shimon has created two songs, using a database of nearly 5,000 songs, including works from Beethoven, Miles Davis and Lady Gaga. This is song two. Credit: Georgia Institute of Technology

Bretan says this is the first time a robot has used to create music. And unlike its days of improvising, when it played monophonically, Shimon is able to play harmonies and chords. It's also thinking much more like a human musician, focusing less on the next note, as it did before, and more on the overall structure of the composition.

"When we play or listen to music, we don't think about the next note and only that next note," said Bretan. "An artist has a bigger idea of what he or she is trying to achieve within the next few measures or later in the piece. Shimon is now coming up with higher-level musical semantics. Rather than thinking note by note, it has a larger idea of what it wants to play as a whole."

Shimon was created by Bretan's advisor, Gil Weinberg, director of Georgia Tech's Center for Music Technology.

Mason Bretan, a Ph.D. candidate in music technology, is the brain behind Shimon, a marimba-playing robot that is writing and playing its own music using deep learning. Credit: Georgia Institute of Technology

"This is a leap in Shimon's musical quality because it's using deep learning to create a more structured and coherent composition," said Weinberg, a professor in the School of Music. "We want to explore whether robots could become musically creative and generate new music that we humans could find beautiful, inspiring and strange."

Shimon will create more pieces in the future. As long as the researchers feed it a different seed, the robot will produce something different each time—music that the researchers can't predict. In the first piece, Bretan fed Shimon a melody comprised of eighth notes. It received a sixteenth note melody the second time, which influenced it to generate faster note sequences.

Bretan acknowledges that he can't pick out individual songs that Shimon is referencing. He is able to recognize classical chord progression and influences of artists, such as Mozart, for example.

"They sound like a fusion of jazz and classical," said Bretan, who plays the keyboards and guitar in his free time. "I definitely hear more classical, especially in the harmony. But then I hear chromatic moving steps in the first piece—that's definitely something you hear in jazz."

A look at the musical notes of Shimon's first two compositions. Credit: Georgia Institute of Technology

Explore further: Music Technology Researchers Create New Robotic Percussionist

Related Stories

Music Technology Researchers Create New Robotic Percussionist

November 7, 2008

(PhysOrg.com) -- Georgia Tech has created an improved version of the robotic percussionist. The second edition, named Shimon, is designed to play a melodic instrument – the marimba. It, therefore, utilizes more sophisticated ...

Astronomer creates music using star oscillations

August 13, 2015

(Phys.org)—Astronomer Burak Ulaş, with the Izmir Turk College Planetarium in Turkey has taken his work into a musical dimension, using star oscillations as a source for a musical composition. He has uploaded a paper describing ...

Recommended for you

New paper answers causation conundrum

November 17, 2017

In a new paper published in a special issue of the Philosophical Transactions of the Royal Society A, SFI Professor Jessica Flack offers a practical answer to one of the most significant, and most confused questions in evolutionary ...

Chance discovery of forgotten 1960s 'preprint' experiment

November 16, 2017

For years, scientists have complained that it can take months or even years for a scientific discovery to be published, because of the slowness of peer review. To cut through this problem, researchers in physics and mathematics ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.