UC Davis team's piano system animates hands to do-re-mi (w/ video)

(Phys.org)—Researchers from the University of California, Davis, have built an algorithm designed to create a computer-generated animation of piano-playing, showing finger placement and wrist motions. The novel part of all this is that the team's system takes an audio file and creates a piano-playing cartoon to match it, using sound to animate the piano-playing performance. The software can automatically generate a 3-D animation of piano-playing hands based entirely on the music input. Each chord in a piece translates into a finger position and hand motion. They have worked out an algorithm that carefully takes the position of every finger into account, even when that finger is not involved in the next chord.

Their work is considered interesting with the understanding that any animation of correct piano-playing is a difficult task, considering constraints on the timing of the right hand motions, complex finger-crossings to sound the right keys and to accomplish varied chords, and, as the researchers said, "the high number of degrees of freedom of the hand and fingers."

The researchers describe their work in their paper published in September in Computer Animation and Virtual Worlds, titled "A system for automatic animation of piano performances." Authors Yuanfeng Zhu, Ajay Sundar Ramakrishnan, Bernd Hamann and Michael Neff said that their paper presents the very first system for automatically generating 3-D animations of piano performance given an input midi music file. They validated their approach by directly comparing actual piano playing and the simulation of a complete music piece that called for different playing skills.

They selected a music piece, Mussorgsky's Childhood Memory, to show how their system supported piano-paying features such as finger crossovers and arpeggio skills.

Basically, their software analyzes a music file and then builds animated hands to play the music. The researchers designed the software to calculate the animated fingers' most efficient paths.

The researchers suggest that their technique is realistic enough to be used to self-teach the piano. "In the field of piano tuition, given any piece of music, amateur players find it difficult to determine the most effective fingering and to render it exactly the way it is supposed to be performed. Therefore, a system capable of generating high-quality piano animation may also be useful in piano tutoring." Piano learners could use a motion-capture camera to record their practice, then compare the data to the program's ultra-relaxed formula for playing the same song to see whether their hands are in the best possible position. The authors also said that it would be possible to apply the animation to other instruments, such as violins and flutes.

Nonetheless, their actual animation appears to fall short of their technique's goal for "high-quality piano animation." Several comments reacting to the video of their work either questioned if those looked like real fingers or said the animated hands looked too creepy for words. The finger renderings have a limp, pancake-batter look and may distract a viewer from concentrating on the system as a tool for instruction.

The authors intend to work on their system further; they said they recognize that the system is not capable of generating "emotional piano playing that reflects a personal understanding of the music and player's performance background...This is the main future work we will be pursuing."

© 2012 Phys.org