Computer app allows classical musicians to perform solo with virtual accompaniment

Musicians can now perform as the soloist with a full philharmonic orchestra from the comfort of their own living rooms, thanks to a new computer system that will be described in a presentation at the 167th meeting of the Acoustical Society of America, to be held May 5-9 in Providence, Rhode Island.

"Classical musicians spend untold hours learning to play the solo literature featuring their instrument, but very few ever perform this music with the accompanying ensemble. The reason is it takes many players to make an orchestra, but only one to be a soloist," said Christopher Raphael, chair of Computer Science in the School of Informatics and Computing at Indiana University, Bloomington, and a former professional oboist. "While the oboe is not the favorite solo instrument of composers or audiences, I have performed as a soloist 10 or 15 times. The experience was thrilling, so I wanted to find a way to replicate the feeling of this experience, and to share it with others," he said.

"I have worked on this for many years," Raphael added, "since it represents a kind a grand challenge—bringing together an application domain I care deeply about, and some areas of computer science in which I have expertise."

In musical accompaniment, or, for that matter, in any ensemble performance, each musician must form ongoing predictions about the way the music will evolve and continually revise these predictions based on what they hear. To emulate this process with a computer model, Raphael developed a so-called Bayesian Belief Network, which is "a simple model for musical timing that understands the nominal note values from the score and what they imply about duration, and the way tempo changes fluidly in a performance," he explained.

To model the hearing of the accompanists—and thus be able to identify, and respond to, the notes played by the soloist, and when they occur—the system uses an algorithm known as a hidden Markov model, which is commonly employed in speech-recognition technologies.

The simulated orchestra is synthesized from a prerecorded orchestra, which means there is no limit to the number of instruments involved, "though it isn't always easy to find a recording of a concerto minus the soloist," Raphael says.

The system—which Raphael has dubbed the "Informatics Philharmonic"—is designed to understand the "imprecise nature" of humans, and, like an artificial intelligence system, can learn to adapt to the soloist's interpretation of the music. The model can be automatically trained from past performances (and must be trained for each individual soloist), "thus capturing the essence of the human rehearsal process in which one learns from example," he said.

Technically, he added, the program has a score of the piece that gives the basic information of the score, more or less as it is presented in the score: pitches and rhythmic values. This is the information that is used to "hear" the soloist. To create the accompaniment, the program uses a prerecorded accompaniment performance that is matched to the symbolic score offline, which allows the accompaniment to be played back while its timing is warped so that it synchronizes with the soloist.