(Phys.org)—Researchers from the University of California, Davis, have built an algorithm designed to create a computer-generated animation of piano-playing, showing finger placement and wrist motions. The novel part of all this is that the team's system takes an audio file and creates a piano-playing cartoon to match it, using sound to animate the piano-playing performance. The software can automatically generate a 3-D animation of piano-playing hands based entirely on the music input. Each chord in a piece translates into a finger position and hand motion. They have worked out an algorithm that carefully takes the position of every finger into account, even when that finger is not involved in the next chord.
Their work is considered interesting with the understanding that any animation of correct piano-playing is a difficult task, considering constraints on the timing of the right hand motions, complex finger-crossings to sound the right keys and to accomplish varied chords, and, as the researchers said, "the high number of degrees of freedom of the hand and ﬁngers."
The researchers describe their work in their paper published in September in Computer Animation and Virtual Worlds, titled "A system for automatic animation of piano performances." Authors Yuanfeng Zhu, Ajay Sundar Ramakrishnan, Bernd Hamann and Michael Neff said that their paper presents the very first system for automatically generating 3-D animations of piano performance given an input midi music file. They validated their approach by directly comparing actual piano playing and the simulation of a complete music piece that called for different playing skills.
They selected a music piece, Mussorgsky's Childhood Memory, to show how their system supported piano-paying features such as finger crossovers and arpeggio skills.
Basically, their software analyzes a music file and then builds animated hands to play the music. The researchers designed the software to calculate the animated fingers' most efficient paths.
The researchers suggest that their technique is realistic enough to be used to self-teach the piano. "In the ﬁeld of piano tuition, given any piece of music, amateur players ﬁnd it difﬁcult to determine the most effective ﬁngering and to render it exactly the way it is supposed to be performed. Therefore, a system capable of generating high-quality piano animation may also be useful in piano tutoring." Piano learners could use a motion-capture camera to record their practice, then compare the data to the program's ultra-relaxed formula for playing the same song to see whether their hands are in the best possible position. The authors also said that it would be possible to apply the animation to other instruments, such as violins and flutes.
Nonetheless, their actual animation appears to fall short of their technique's goal for "high-quality piano animation." Several comments reacting to the video of their work either questioned if those looked like real fingers or said the animated hands looked too creepy for words. The finger renderings have a limp, pancake-batter look and may distract a viewer from concentrating on the system as a tool for instruction.
The authors intend to work on their system further; they said they recognize that the system is not capable of generating "emotional piano playing that reﬂects a personal understanding of the music and player's performance background...This is the main future work we will be pursuing."
Explore further: MIT groups develop smartphone system THAW that allows for direct interaction between devices
More information: A system for automatic animation of piano performances, Computer Animation and Virtual Worlds, DOI: 10.1002/cav.1477 (PDF link)
Playing the piano requires one to precisely position one's hand in order to strike particular combinations of keys at specific moments in time. This paper presents the first system for automatically generating three-dimensional animations of piano performance, given an input midi music file. A graph theory-based motion planning method is used to decide which set of fingers should strike the piano keys for each chord. As the progression of the music is anticipated, the positions of unused fingers are calculated to make possible efficient fingering of future notes. Initial key poses of the hands, including those for complex piano techniques such as crossovers and arpeggio, are determined on the basis of the finger positions and piano theory. An optimization method is used to refine these poses, producing a natural and minimal energy pose sequence. Motion transitions between poses are generated using a combination of sampled piano playing motion and music features, allowing the system to support different playing styles. Our approach is validated through direct comparison with actual piano playing and simulation of a complete music piece requiring various playing skills. Extensions of our system are discussed.