Computer scientists synthesize cloth sounds for animation

Sep 26, 2012 by Bill Steele
Computer scientists synthesize cloth sounds for animation
To match sound to action, a computer looks at frames of computer animation of a piece of fabric and measures how much parts of the image are raised or lowered as a basis for synthesizing "crumpling" sound.

(Phys.org)—Someday, virtual reality may be so well done that you won't be able to tell it's just computer animation. To make that happen we'll have to hear it as well as see it, from dramatic noises like the sound of a breaking glass to subtle details like the rustling of clothing as characters move.

Cornell researchers who specialize in creating sound for are adding the sounds of cloth to their repertoire. Doug James and Steve Marschner, associate professors of , and graduate student Steven An reported their method of synthesizing cloth sounds at the SIGGRAPH conference Aug. 7 in Los Angeles.

Soundtracks for animated movies are still mostly constructed by hand, manually matching recorded sounds to the action. But it's not always possible to find appropriate sounds, and in interactive virtual environments you won't always know ahead of time what the action will be. One solution is to synthesize sound based on the physics of what's happening in a scene.

This video is not supported by your browser at this time.
First person view: Using binaural recordings, realistic sounds of a person's clothing can be added to enhance virtual reality.

The Cornell researchers built a system that uses digitally synthesized sound as a sort of to insert real recorded sounds. They synthesize an approximate sound based on two characteristics of the action: , as the cloth moves against itself or other objects, and crumpling, as the cloth deforms. To teach their system to synthesize these sounds, the researchers made controlled recordings of friction sounds by spinning a cloth-covered roller while holding a piece of cloth against it, and crumpling sounds by manipulating a piece of cloth while avoiding any sliding contact.

But the computer-synthesized sounds still aren't very realistic, so the "target" sound is broken up into microsecond chunks, which the computer matches against a database of similar chunks of recorded real cloth sounds which are then reassembled smoothly to produce a convincing soundtrack. Binaural recordings were included to produce "first-person" sounds for a viewer wearing stereo headphones.

A human sound designer oversees the process, choosing the type of fabric and tweaking the rules by which sounds are matched.

Computer scientists synthesize cloth sounds for animation
To create a model of friction sounds, researchers hold a piece of cloth against a motorized cloth roller. Ultimately, rotor speed will be matched to the speed of movement in an animation.

In part, James said, the approach was inspired by speech synthesis. "People thought they could synthesize speech using oscillator models," James explained, "but it didn't sound realistic, sort of like 'Speak & Spell,' so they ended up concatenating bits of recorded human speech for realism."

As demonstrations, the researchers created the sounds of cotton and polyester sheets being spread over a couch, a person exercising in a nylon windbreaker and corduroy pants, and the same person boxing from a first-person viewpoint.

The computation is still time-consuming, so it's not ready for real-time virtual environments yet. Getting the ideal sound still involves trial and error, the researchers reported. "This is a proof of concept," James said. "One next step would be optimizing the method to make it run in real time."

Practical applications may require a more elaborate database of cloth sounds, and still to be added are impact and tension sounds like those of a flag whipping in the wind.

Explore further: Newest computer neural networks can identify visual objects as well as the primate brain

Related Stories

Computers synthesize sounds to go with graphics

Aug 09, 2011

Computer-generated imagery usually relies on recorded sound to complete the illusion. Recordings can, however, limit the range of sounds you can produce, especially in future virtual reality environments where ...

Scientists make animated collisions sounds realistic

Aug 21, 2012

(Phys.org) -- Bang. Clatter. Tinkle. Jingle. When solid objects collide in the real world we hear a sharp impact sound, sometimes followed by a ringing aftershock. Creating sounds like that to accompany computer ...

Researchers bring noise to virtual worlds

Oct 27, 2009

(PhysOrg.com) -- Computer scientists have developed a method to synthesize the sounds of cymbals, falling garbage cans and lids, and plastic water-cooler bottles and recycling bins.

Sounds of northern lights are born close to ground

Jul 09, 2012

For the first time, researchers at Aalto University in Finland have located where the sounds associated with the northern lights are created. The auroral sounds that have been described in folktales and by wilderness wanderers ...

Recommended for you

Cloud computing helps make sense of cloud forests

Dec 17, 2014

The forests that surround Campos do Jordao are among the foggiest places on Earth. With a canopy shrouded in mist much of time, these are the renowned cloud forests of the Brazilian state of São Paulo. It is here that researchers ...

Teaching robots to see

Dec 15, 2014

Syed Saud Naqvi, a PhD student from Pakistan, is working on an algorithm to help computer programmes and robots to view static images in a way that is closer to how humans see.

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.