Confining the data explosion

“Let’s talk about the weather,” says Clemens-August Thole of the Fraunhofer Institute for Algorithms and Scientific Computing SCAI in St. Augustin. “As of 2008 Germany’s national meteorological service, the Deutscher Wetterdienst, will each day produce about 20 forecasts of the weather for the next few days. That will generate about 2.5 petabytes of data a year.”

One petabyte is equivalent to something like 2,000 hard disks, each with 500 gigabytes of memory. To diminish this volume, Thole and his team have developed a software program that reduces meteorological data in GRIB format by a factor of 2.5.

“Germany is covered with a network of grid points for these simulations,“ explains Prof. Rudolph Lorentz. “The results of simulations at neighboring points are not independent of one another. All we need is a good prediction of the values at any grid point in order to reduce the data volume significantly“.

Simulations have become indispensable in the automobile industry, too. However, even a virtual crash takes up several gigabytes of memory. “On the computer, any one of these models crashes into the wall 100 to 150 times a day,” says Rodrigo Iza-Teran. That quickly adds up to 100 terabytes or more of data. Prior to compression, the development engineers specify the required accuracy of the stored data, for example one millimeter in the case of vehicle crashes. In making this quantization, they are deliberately accepting a loss of data. All other stages of compression are loss-free, which means the data can be reconstructed 1:1.

The scientists at SCAI can offer compression factors of 10, 7 or 5, depending on the needs of the users. “Our approach is rather like video coding,” says Thole. “One of the tricks is to find out which components move without deforming. Another is to predict the changing content as accurately as possible.”

Nearly all German car manufacturers use the software, and with good reason: They can save valuable memory space, and the exchange of data runs smoothly. What is more, compressed data can be visualized faster and more easily. The decompression software has meanwhile been integrated in many post-processors, enabling files to be opened without the intermediate step of unzipping them.

Rodrigo Iza-Teran, Prof. Dr. Rudolph Lorentz and Clemens-August Thole have been awarded the Joseph-von-Fraunhofer Prize 2007 for their work on the compression of simulation results. The jury was impressed not only by the outstanding mathematical methods employed, but also by the great success that the compression method has scored with users.

Source: Fraunhofer-Gesellschaft