Confining the data explosion

Oct 19, 2007

More simulation also requires more space on the hard disk – no prob-lem for the new compression software. The volume of data for crash and weather simulations is shrunk to as little as a tenth of its original size.

“Let’s talk about the weather,” says Clemens-August Thole of the Fraunhofer Institute for Algorithms and Scientific Computing SCAI in St. Augustin. “As of 2008 Germany’s national meteorological service, the Deutscher Wetterdienst, will each day produce about 20 forecasts of the weather for the next few days. That will generate about 2.5 petabytes of data a year.”

One petabyte is equivalent to something like 2,000 hard disks, each with 500 gigabytes of memory. To diminish this volume, Thole and his team have developed a software program that reduces meteorological data in GRIB format by a factor of 2.5.

“Germany is covered with a network of grid points for these simulations,“ explains Prof. Rudolph Lorentz. “The results of simulations at neighboring points are not independent of one another. All we need is a good prediction of the values at any grid point in order to reduce the data volume significantly“.

Simulations have become indispensable in the automobile industry, too. However, even a virtual crash takes up several gigabytes of memory. “On the computer, any one of these models crashes into the wall 100 to 150 times a day,” says Rodrigo Iza-Teran. That quickly adds up to 100 terabytes or more of data. Prior to compression, the development engineers specify the required accuracy of the stored data, for example one millimeter in the case of vehicle crashes. In making this quantization, they are deliberately accepting a loss of data. All other stages of compression are loss-free, which means the data can be reconstructed 1:1.

The scientists at SCAI can offer compression factors of 10, 7 or 5, depending on the needs of the users. “Our approach is rather like video coding,” says Thole. “One of the tricks is to find out which components move without deforming. Another is to predict the changing content as accurately as possible.”

Nearly all German car manufacturers use the software, and with good reason: They can save valuable memory space, and the exchange of data runs smoothly. What is more, compressed data can be visualized faster and more easily. The decompression software has meanwhile been integrated in many post-processors, enabling files to be opened without the intermediate step of unzipping them.

Rodrigo Iza-Teran, Prof. Dr. Rudolph Lorentz and Clemens-August Thole have been awarded the Joseph-von-Fraunhofer Prize 2007 for their work on the compression of simulation results. The jury was impressed not only by the outstanding mathematical methods employed, but also by the great success that the compression method has scored with users.

Source: Fraunhofer-Gesellschaft

Explore further: Modeling the ripples of health care information

add to favorites email to friend print save as pdf

Related Stories

Poor-man's supercomputing goes commercial

Dec 17, 2008

(PhysOrg.com) -- Grid computing technology has long been the darling of cash-strapped academics in desperate need of raw processing power. Now a groundbreaking European research effort has created an industrial-strength platform ...

Recommended for you

Forging a photo is easy, but how do you spot a fake?

Nov 21, 2014

Faking photographs is not a new phenomenon. The Cottingley Fairies seemed convincing to some in 1917, just as the images recently broadcast on Russian television, purporting to be satellite images showin ...

Algorithm, not live committee, performs author ranking

Nov 21, 2014

Thousands of authors' works enter the public domain each year, but only a small number of them end up being widely available. So how to choose the ones taking center-stage? And how well can a machine-learning ...

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

Reaper6971
not rated yet Oct 19, 2007
"500 gigabytes of memory"

Don't you mean Storage Capacity? grr...

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.