Confining the data explosion

Oct 19, 2007

More simulation also requires more space on the hard disk – no prob-lem for the new compression software. The volume of data for crash and weather simulations is shrunk to as little as a tenth of its original size.

“Let’s talk about the weather,” says Clemens-August Thole of the Fraunhofer Institute for Algorithms and Scientific Computing SCAI in St. Augustin. “As of 2008 Germany’s national meteorological service, the Deutscher Wetterdienst, will each day produce about 20 forecasts of the weather for the next few days. That will generate about 2.5 petabytes of data a year.”

One petabyte is equivalent to something like 2,000 hard disks, each with 500 gigabytes of memory. To diminish this volume, Thole and his team have developed a software program that reduces meteorological data in GRIB format by a factor of 2.5.

“Germany is covered with a network of grid points for these simulations,“ explains Prof. Rudolph Lorentz. “The results of simulations at neighboring points are not independent of one another. All we need is a good prediction of the values at any grid point in order to reduce the data volume significantly“.

Simulations have become indispensable in the automobile industry, too. However, even a virtual crash takes up several gigabytes of memory. “On the computer, any one of these models crashes into the wall 100 to 150 times a day,” says Rodrigo Iza-Teran. That quickly adds up to 100 terabytes or more of data. Prior to compression, the development engineers specify the required accuracy of the stored data, for example one millimeter in the case of vehicle crashes. In making this quantization, they are deliberately accepting a loss of data. All other stages of compression are loss-free, which means the data can be reconstructed 1:1.

The scientists at SCAI can offer compression factors of 10, 7 or 5, depending on the needs of the users. “Our approach is rather like video coding,” says Thole. “One of the tricks is to find out which components move without deforming. Another is to predict the changing content as accurately as possible.”

Nearly all German car manufacturers use the software, and with good reason: They can save valuable memory space, and the exchange of data runs smoothly. What is more, compressed data can be visualized faster and more easily. The decompression software has meanwhile been integrated in many post-processors, enabling files to be opened without the intermediate step of unzipping them.

Rodrigo Iza-Teran, Prof. Dr. Rudolph Lorentz and Clemens-August Thole have been awarded the Joseph-von-Fraunhofer Prize 2007 for their work on the compression of simulation results. The jury was impressed not only by the outstanding mathematical methods employed, but also by the great success that the compression method has scored with users.

Source: Fraunhofer-Gesellschaft

Explore further: Computer program to take on world's best in Texas Hold 'em

Related Stories

Certain interactive tools click with web users

15 minutes ago

Before web developers add the newest bells and the latest whistles to their website designs, a team of researchers suggests they zoom in on the tools that click with the right users and for the right tasks.

Printing silicon on paper, with lasers

34 minutes ago

In seeking to develop the next generation of micro-electronic transistors, researchers have long sought to find the next best thing to replace silicon. To this end, a wealth of recent research into fully ...

Image: Claude Nicollier repairing Hubble

55 minutes ago

This week marks the 25th anniversary of the launch of the Hubble space telescope. Hubble was designed to be maintained and repaired by astronauts and since its launch in 1990 five Space Shuttle missions made ...

Recommended for you

Ears, grips and fists take on mobile phone user ID

10 hours ago

A research project has been under way to explore a biometric authentication system dubbed Bodyprint, with interesting test results. Bodyprint has been designed to detect users' biometric features using the ...

Russian hackers read Obama emails, report says

21 hours ago

Emails to and from President Barack Obama were read by Russian hackers last year in a breach of the White House's unclassified computer system, The New York Times said Saturday.

Supermarkets welcome cold-comfort edge of F1 aerofoils

Apr 25, 2015

UK-based Williams Advanced Engineering, the technology and engineering services business of the Williams Group, has collaborated with UK-based Aerofoil Energy to develop an aerodynamic device that can reduce ...

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

Reaper6971
not rated yet Oct 19, 2007
"500 gigabytes of memory"

Don't you mean Storage Capacity? grr...

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.