New tool enables powerful data analysis

Jan 08, 2009
After using the algorithm to determine the filament structure of an aerogel -- a lightweight foam used in shielding electronic equipment in satellites -- the researchers were able to compute changes to its structural integrity by the simulated impact of a micrometeorite traveling at 10,000 miles per hour (red sphere on left). Image: Attila Gyulassy/UC Davis Copyright UC Regents

(PhysOrg.com) -- A powerful computing tool that allows scientists to extract features and patterns from enormously large and complex sets of raw data has been developed by scientists at University of California, Davis, and Lawrence Livermore National Laboratory. The tool - a set of problem-solving calculations known as an algorithm - is compact enough to run on computers with as little as two gigabytes of memory.

The team that developed this algorithm has already used it to probe a slew of phenomena represented by billions of data points, including analyzing and creating images of flame surfaces; searching for clusters and voids in a virtual universe experiment; and identifying and tracking pockets of fluid in a simulated mixing of two fluids.

"What we've developed is a workable system of handling any data in any dimension," said Attila Gyulassy, who led the five-year development effort while pursuing a PhD in computer science at UC Davis. "We expect this algorithm will become an integral part of a scientist's toolbox to answer questions about data."

A paper describing the new algorithm was published in the November-December issue of IEEE Transactions on Visualization and Computer Graphics.

This image of an early moment in the simulated mixing of two fluids was created by researchers using a powerful new algorithm they developed to extract features and patterns from massive data sets. In the image, blue and red spheres and the lines between them represent the branching of pockets of fluid. Image: Attila Gyulassi/UC Davis copyright UC Regents


Computers are widely used to perform simulations of real-world phenomena and to capture results of physical experiments and observations, storing this information as collections of numbers. But as the size of these data sets has burgeoned, hand-in-hand with computer capacity, analysis has grown increasingly difficult.

A mathematical tool to extract and visualize useful features from data sets has existed for nearly 40 years - in theory. Called the Morse-Smale complex, it partitions sets by similarity of features and encodes them into mathematical terms. But working with the Morse-Smale complex is not easy. "It's a powerful language. But a cost of that, is that using it meaningfully for practical applications is very difficult," Gyulassy said.

Gyulassy's algorithm divides data sets into parcels of cells, then analyzes each parcel separately using the Morse-Smale complex. Results of those computations are then merged together. As new parcels are created from merged parcels, they are analyzed and merged yet again. At each step, data that do not need to be stored in memory are discarded, drastically reducing the computing power required to run the calculations.

One of Gyulassy's tests of the algorithm was to use it to analyze and track the formation and movement of pockets of fluid in the simulated mixing of two fluids: one dense, one light. The complexity of this data set is so vast - it consists of more than one billion data points on a three-dimensional grid - it challenges even supercomputers, Gyulassy said. Yet the new algorithm with its streamlining features was able to perform the analysis on a laptop computer with just two gigabytes of memory. Although Gyulassy had to wait nearly 24 hours for the little machine to complete its calculations, at the end of this process he could pull up images in mere seconds to illustrate phenomena he was interested in, such as the branching of fluid pockets in the mixture.

Two main factors are driving the need for analysis of large data sets, said co-author Bernd Hamann: a surge in the use of powerful computers that can produce huge amounts of data, and an upswing in affordability and availability of sensing devices that researchers deploy in the field and lab to collect a profusion of data.

"Our data files are becoming larger and larger, while the scientist has less and less time to understand them," said Hamann, a professor of computer science and associate vice chancellor for research at UC Davis. "But what are the data good for if we don't have the means of applying mathematically sound and computationally efficient computer analysis tools to look for what is captured in them?"

Gyulassy is currently developing software that will allow others to put the algorithm to use. He expects the learning curve to be steep for this open-source product, "but if you just learn the minimal amount about what a Morse-Smale complex is," he said, "it will be pretty intuitive."

Source: University of California - Davis

Explore further: System to automatically find a common type of programming bug significantly outperforms its predecessors

add to favorites email to friend print save as pdf

Related Stories

Fitness app connects exercisers to experts

Mar 24, 2015

Can advanced networking and next-generation applications help solve some of our nation's most pressing health problems? Can mobile devices and high-speed Internet be used to improve our health and well-being? ...

Zoo innovations has animals foraging for food

Mar 20, 2015

When red pandas go on exhibit for the first time at Brookfield Zoo in July, they'll be housed around a broad tree that looks like a giant bonsai and has magical qualities.

Musical sparks fly with new electrical innovation

Mar 17, 2015

An engineering student from Plymouth University has given a 19th century electrical device a modern twist, using it to generate striking new versions of some of Hollywood's best loved film scores.

Researchers rethink how our feathered friends evolved

Mar 17, 2015

A recently published global genome study that used the data-intensive Gordon supercomputer at the San Diego Supercomputer at the University of California, San Diego, has researchers rethinking how avian lineages ...

Recommended for you

Blue Freedom uses power of flowing water to charge

8 minutes ago

Good friends may decide to tell you something that is not true but nonetheless sustaining: Nothing is impossible. That was the case of Blue Freedom co-founder who asked his friend if it would be possible ...

Special ops troops using flawed intel software

28 minutes ago

Special operations troops heading to war zones are asking for commercial intelligence analysis software they say will help their missions. But their requests are languishing, and they are being ordered to use a flawed, in-house ...

Applications of optical fibre for sensors

4 hours ago

Mikel Bravo-Acha's PhD thesis has focused on the applications of optical fibre as a sensor. In the course of his research, conducted at the NUP/UPNA-Public University of Navarre, he monitored a sensor fitted to optical fibre ...

Twitter chief vows to help Indonesia fight disasters

4 hours ago

Twitter chief Dick Costolo said Thursday the microblogging site planned to work with Indonesian authorities to warn people about natural disasters that regularly hit the archipelago, from earthquakes to volcanic ...

Virtual robotization for human limbs

5 hours ago

Recent advances in computer gaming technology allow for an increasingly immersive gaming experience. Gesture input devices, for example, synchronise a player's actions with the character on the screen. Entertainment ...

User comments : 4

Adjust slider to filter visible comments by rank

Display comments: newest first

googleplex
not rated yet Jan 08, 2009
This sounds almost too good to be true. Anyone got a link to the paper?
earls
not rated yet Jan 08, 2009
jrdewoodjr
not rated yet Jan 08, 2009
i'm going to use this as a primary story element> very slick resonant flow curls!
kepler
not rated yet Jan 11, 2009
http://ieeexplore...=4658183&isnumber=4658121
$29 for non-members

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.