UC Davis team's piano system animates hands to do-re-mi (w/ video)

Oct 01, 2012 by Nancy Owano report
The axes in researchers' model. Credit: : Zhu, Yuanfeng et al.

(Phys.org)—Researchers from the University of California, Davis, have built an algorithm designed to create a computer-generated animation of piano-playing, showing finger placement and wrist motions. The novel part of all this is that the team's system takes an audio file and creates a piano-playing cartoon to match it, using sound to animate the piano-playing performance. The software can automatically generate a 3-D animation of piano-playing hands based entirely on the music input. Each chord in a piece translates into a finger position and hand motion. They have worked out an algorithm that carefully takes the position of every finger into account, even when that finger is not involved in the next chord.

Their work is considered interesting with the understanding that any animation of correct -playing is a difficult task, considering constraints on the timing of the right , complex finger-crossings to sound the right keys and to accomplish varied chords, and, as the researchers said, "the high number of of the hand and fingers."

This video is not supported by your browser at this time.
Credit: Computer Animation and Virtual Worlds, DOI: 10.1002/cav.1477

The researchers describe their work in their paper published in September in and Virtual Worlds, titled "A system for automatic animation of piano performances." Authors Yuanfeng Zhu, Ajay Sundar Ramakrishnan, Bernd Hamann and Michael Neff said that their paper presents the very first system for automatically generating 3-D animations of piano performance given an input midi music file. They validated their approach by directly comparing actual piano playing and the simulation of a complete music piece that called for different playing skills.

They selected a music piece, Mussorgsky's Childhood Memory, to show how their system supported piano-paying features such as finger crossovers and arpeggio skills.

Basically, their software analyzes a music file and then builds animated hands to play the music. The researchers designed the software to calculate the animated fingers' most efficient paths.

The researchers suggest that their technique is realistic enough to be used to self-teach the piano. "In the field of piano tuition, given any piece of music, amateur players find it difficult to determine the most effective fingering and to render it exactly the way it is supposed to be performed. Therefore, a system capable of generating high-quality piano animation may also be useful in piano tutoring." Piano learners could use a motion-capture camera to record their practice, then compare the data to the program's ultra-relaxed formula for playing the same song to see whether their hands are in the best possible position. The authors also said that it would be possible to apply the animation to other instruments, such as violins and flutes.

Nonetheless, their actual animation appears to fall short of their technique's goal for "high-quality piano animation." Several comments reacting to the video of their work either questioned if those looked like real fingers or said the animated hands looked too creepy for words. The finger renderings have a limp, pancake-batter look and may distract a viewer from concentrating on the system as a tool for instruction.

The authors intend to work on their system further; they said they recognize that the system is not capable of generating "emotional piano playing that reflects a personal understanding of the and player's performance background...This is the main future work we will be pursuing."

Explore further: Computer-assisted accelerator design

More information: A system for automatic animation of piano performances, Computer Animation and Virtual Worlds, DOI: 10.1002/cav.1477 (PDF link)

Playing the piano requires one to precisely position one's hand in order to strike particular combinations of keys at specific moments in time. This paper presents the first system for automatically generating three-dimensional animations of piano performance, given an input midi music file. A graph theory-based motion planning method is used to decide which set of fingers should strike the piano keys for each chord. As the progression of the music is anticipated, the positions of unused fingers are calculated to make possible efficient fingering of future notes. Initial key poses of the hands, including those for complex piano techniques such as crossovers and arpeggio, are determined on the basis of the finger positions and piano theory. An optimization method is used to refine these poses, producing a natural and minimal energy pose sequence. Motion transitions between poses are generated using a combination of sampled piano playing motion and music features, allowing the system to support different playing styles. Our approach is validated through direct comparison with actual piano playing and simulation of a complete music piece requiring various playing skills. Extensions of our system are discussed.

Related Stories

Learn to play by playing Songs2See

Aug 27, 2010

Recorder, guitar, piano or violin - many children and young people learn to play these popular instruments. It requires a lot of practice to read note after note from the sheet music and then strike the right ...

Recommended for you

Computer-assisted accelerator design

Apr 22, 2014

Stephen Brooks uses his own custom software tool to fire electron beams into a virtual model of proposed accelerator designs for eRHIC. The goal: Keep the cost down and be sure the beams will circulate in ...

First steps towards "Experimental Literature 2.0"

Apr 21, 2014

As part of a student's thesis, the Laboratory of Digital Humanities at EPFL has developed an application that aims at rearranging literary works by changing their chapter order. "The human simulation" a saga ...

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Oct 01, 2012
At last, I can master Chopsticks!

More news stories

Robot scouts rooms people can't enter

(Phys.org) —Firefighters, police officers and military personnel are often required to enter rooms with little information about what dangers might lie behind the door. A group of engineering students at ...

Finalists named in Bloomberg European city contest

Amsterdam wants to create an online game to get unemployed young people engaged in finding jobs across Europe. Schaerbeek, Belgium, envisions using geothermal mapping to give households personalized rundowns of steps to save ...

Internet TV case: US justices skeptical, concerned

Grappling with fast-changing technology, U.S. Supreme Court justices debated Tuesday whether they can protect the copyrights of TV broadcasters to the shows they send out without strangling innovations in ...

Brazil passes trailblazing Internet privacy law

Brazil's Congress on Tuesday passed comprehensive legislation on Internet privacy in what some have likened to a web-user's bill of rights, after stunning revelations its own president was targeted by US ...

In the 'slime jungle' height matters

(Phys.org) —In communities of microbes, akin to 'slime jungles', cells evolve not just to grow faster than their rivals but also to push themselves to the surface of colonies where they gain the best access ...

New alfalfa variety resists ravenous local pest

(Phys.org) —Cornell plant breeders have released a new alfalfa variety with some resistance against the alfalfa snout beetle, which has ravaged alfalfa fields in nine northern New York counties and across ...