UC Davis team's piano system animates hands to do-re-mi (w/ video)

Oct 01, 2012 by Nancy Owano report
The axes in researchers' model. Credit: : Zhu, Yuanfeng et al.

(Phys.org)—Researchers from the University of California, Davis, have built an algorithm designed to create a computer-generated animation of piano-playing, showing finger placement and wrist motions. The novel part of all this is that the team's system takes an audio file and creates a piano-playing cartoon to match it, using sound to animate the piano-playing performance. The software can automatically generate a 3-D animation of piano-playing hands based entirely on the music input. Each chord in a piece translates into a finger position and hand motion. They have worked out an algorithm that carefully takes the position of every finger into account, even when that finger is not involved in the next chord.

Their work is considered interesting with the understanding that any animation of correct -playing is a difficult task, considering constraints on the timing of the right , complex finger-crossings to sound the right keys and to accomplish varied chords, and, as the researchers said, "the high number of of the hand and fingers."

This video is not supported by your browser at this time.
Credit: Computer Animation and Virtual Worlds, DOI: 10.1002/cav.1477

The researchers describe their work in their paper published in September in and Virtual Worlds, titled "A system for automatic animation of piano performances." Authors Yuanfeng Zhu, Ajay Sundar Ramakrishnan, Bernd Hamann and Michael Neff said that their paper presents the very first system for automatically generating 3-D animations of piano performance given an input midi music file. They validated their approach by directly comparing actual piano playing and the simulation of a complete music piece that called for different playing skills.

They selected a music piece, Mussorgsky's Childhood Memory, to show how their system supported piano-paying features such as finger crossovers and arpeggio skills.

Basically, their software analyzes a music file and then builds animated hands to play the music. The researchers designed the software to calculate the animated fingers' most efficient paths.

The researchers suggest that their technique is realistic enough to be used to self-teach the piano. "In the field of piano tuition, given any piece of music, amateur players find it difficult to determine the most effective fingering and to render it exactly the way it is supposed to be performed. Therefore, a system capable of generating high-quality piano animation may also be useful in piano tutoring." Piano learners could use a motion-capture camera to record their practice, then compare the data to the program's ultra-relaxed formula for playing the same song to see whether their hands are in the best possible position. The authors also said that it would be possible to apply the animation to other instruments, such as violins and flutes.

Nonetheless, their actual animation appears to fall short of their technique's goal for "high-quality piano animation." Several comments reacting to the video of their work either questioned if those looked like real fingers or said the animated hands looked too creepy for words. The finger renderings have a limp, pancake-batter look and may distract a viewer from concentrating on the system as a tool for instruction.

The authors intend to work on their system further; they said they recognize that the system is not capable of generating "emotional piano playing that reflects a personal understanding of the and player's performance background...This is the main future work we will be pursuing."

Explore further: Modeling the ripples of health care information

More information: A system for automatic animation of piano performances, Computer Animation and Virtual Worlds, DOI: 10.1002/cav.1477 (PDF link)

ABSTRACT
Playing the piano requires one to precisely position one's hand in order to strike particular combinations of keys at specific moments in time. This paper presents the first system for automatically generating three-dimensional animations of piano performance, given an input midi music file. A graph theory-based motion planning method is used to decide which set of fingers should strike the piano keys for each chord. As the progression of the music is anticipated, the positions of unused fingers are calculated to make possible efficient fingering of future notes. Initial key poses of the hands, including those for complex piano techniques such as crossovers and arpeggio, are determined on the basis of the finger positions and piano theory. An optimization method is used to refine these poses, producing a natural and minimal energy pose sequence. Motion transitions between poses are generated using a combination of sampled piano playing motion and music features, allowing the system to support different playing styles. Our approach is validated through direct comparison with actual piano playing and simulation of a complete music piece requiring various playing skills. Extensions of our system are discussed.

Related Stories

Learn to play by playing Songs2See

Aug 27, 2010

Recorder, guitar, piano or violin - many children and young people learn to play these popular instruments. It requires a lot of practice to read note after note from the sheet music and then strike the right ...

Recommended for you

Forging a photo is easy, but how do you spot a fake?

Nov 21, 2014

Faking photographs is not a new phenomenon. The Cottingley Fairies seemed convincing to some in 1917, just as the images recently broadcast on Russian television, purporting to be satellite images showin ...

Algorithm, not live committee, performs author ranking

Nov 21, 2014

Thousands of authors' works enter the public domain each year, but only a small number of them end up being widely available. So how to choose the ones taking center-stage? And how well can a machine-learning ...

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

mrtea
not rated yet Oct 01, 2012
At last, I can master Chopsticks!

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.