Confining the data explosion

October 19, 2007

More simulation also requires more space on the hard disk – no prob-lem for the new compression software. The volume of data for crash and weather simulations is shrunk to as little as a tenth of its original size.

“Let’s talk about the weather,” says Clemens-August Thole of the Fraunhofer Institute for Algorithms and Scientific Computing SCAI in St. Augustin. “As of 2008 Germany’s national meteorological service, the Deutscher Wetterdienst, will each day produce about 20 forecasts of the weather for the next few days. That will generate about 2.5 petabytes of data a year.”

One petabyte is equivalent to something like 2,000 hard disks, each with 500 gigabytes of memory. To diminish this volume, Thole and his team have developed a software program that reduces meteorological data in GRIB format by a factor of 2.5.

“Germany is covered with a network of grid points for these simulations,“ explains Prof. Rudolph Lorentz. “The results of simulations at neighboring points are not independent of one another. All we need is a good prediction of the values at any grid point in order to reduce the data volume significantly“.

Simulations have become indispensable in the automobile industry, too. However, even a virtual crash takes up several gigabytes of memory. “On the computer, any one of these models crashes into the wall 100 to 150 times a day,” says Rodrigo Iza-Teran. That quickly adds up to 100 terabytes or more of data. Prior to compression, the development engineers specify the required accuracy of the stored data, for example one millimeter in the case of vehicle crashes. In making this quantization, they are deliberately accepting a loss of data. All other stages of compression are loss-free, which means the data can be reconstructed 1:1.

The scientists at SCAI can offer compression factors of 10, 7 or 5, depending on the needs of the users. “Our approach is rather like video coding,” says Thole. “One of the tricks is to find out which components move without deforming. Another is to predict the changing content as accurately as possible.”

Nearly all German car manufacturers use the software, and with good reason: They can save valuable memory space, and the exchange of data runs smoothly. What is more, compressed data can be visualized faster and more easily. The decompression software has meanwhile been integrated in many post-processors, enabling files to be opened without the intermediate step of unzipping them.

Rodrigo Iza-Teran, Prof. Dr. Rudolph Lorentz and Clemens-August Thole have been awarded the Joseph-von-Fraunhofer Prize 2007 for their work on the compression of simulation results. The jury was impressed not only by the outstanding mathematical methods employed, but also by the great success that the compression method has scored with users.

Source: Fraunhofer-Gesellschaft

Related Stories

Recommended for you

Microsoft describes hard-to-mimic authentication gesture

August 1, 2015

Photos. Messages. Bank account codes. And so much more—sit on a person's mobile device, and the question is, how to secure them without having to depend on lengthy password codes of letters and numbers. Vendors promoting ...

Power grid forecasting tool reduces costly errors

July 30, 2015

Accurately forecasting future electricity needs is tricky, with sudden weather changes and other variables impacting projections minute by minute. Errors can have grave repercussions, from blackouts to high market costs. ...

Netherlands bank customers can get vocal on payments

August 1, 2015

Are some people fed up with remembering and using passwords and PINs to make it though the day? Those who have had enough would prefer to do without them. For mobile tasks that involve banking, though, it is obvious that ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

Reaper6971
not rated yet Oct 19, 2007
"500 gigabytes of memory"

Don't you mean Storage Capacity? grr...

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.