Energy efficiency and miniaturization thanks to water-cooled chips

New water-cooling technologies that wick off heat right where it is being generated—directly at the chip—offer a promising route to boost significantly the overall energy efficiency of computers. At this year's CeBIT, IBM is presenting its first so-called hot-water-cooled systems, which will provide a sneak preview of future innovations: Supercomputers the size of sugar cubes.

First successful projects

Cooling with hot water combines several advantages: It eliminates the need for energy-intensive coolers, which greatly reduces overall energy consumption. In addition, the removed heat can be reused directly for such purposes as to heat buildings or as process heat. This will improve the CO₂ balance of computers and data centers significantly compared to similar air-cooled systems.

In a pilot project, IBM scientists and engineers from Zurich, Switzerland, and Böblingen, Germany, built a revolutionary hot-water-cooled supercomputer for ETH Zurich, a world-renowned technical university. Dubbed Aquasar, the novel supercomputer is cooled with hot water instead of cold air, and the dissipated heat is used to heat the ETH campus. The innovative system consumes up to 40% less energy than a comparable air-cooled computer. And the CO₂ balance is impressive: By directly using waste heat, the system reduces net emissions by up to 85%.

This is made possible by an innovative cooling system and powerful microchannel coolers attached to the back of the processors. Computer chips, which develop ten times more heat per square centimeter than a hotplate, can thus be cooled efficiently even with 60°C hot water. The entire cooling system of the computer is a closed, hermetically sealed circuit, enabling valuable waste heat to be recovered. Up to 80% of that waste heat can be repurposed by a heat exchanger that delivers it to a second, external heat cycle. In the case of Aquasar at ETH Zurich, the waste heat is fed into the campus heating system. Dr. Ingmar Meijer, Aquasar project manager at IBM Research - Zurich, explains: "With Aquasar we reached an important milestone in the development of low-energy and CO₂-neutral data centers. This sends an important signal to the industry."

The next hot-water-cooled IBM system is already on the drawing board, this time in Germany. It will be significantly larger than Aquasar and is expected to go into operation at the Leibniz Supercomputing Centre (LRZ) in Munich, Germany, by 2012. Called SuperMUC, this new computer will be part of the Partnership for Advanced Computing in Europe (PRACE) HPC infrastructure and made available to scientists and research institutes throughout Europe. The system has a peak performance of 3 petaflop/s (10¹⁵ arithmetic operations per second) and is based on an IBM System X iDataPlex, which contains more than 14,000 Intel Xeon next-generation processors. SuperMUC will be more powerful than 110,000 PCs, enabling LRZ scientists to verify theories, develop experiments and predict results to an unprecedented extent—all this, while still requiring massively less energy.

Dr. Michael Malms, Director of Open Systems and High Performance Computing at the IBM Research and Development Center in Böblingen, Germany, states: "With the installation of SuperMUC we are taking the first step of research towards a system that can be used equally by academia and businesses. It is energy- and cost-effective as well as flexible, which makes it useful for many applications."

Future innovation: 3D integration

Looking further ahead, so-called 3D chips promise even higher performance with lower energy consumption. Paving the way for exascale computers, IBM scientists are pursuing extensive research on 3D integration. 3D chip architectures, in which processors are stacked on top of each other, not only reduce the surface area of the chip but also shorten the communication distance between the chips and increase the bandwidth for data transmission on the chip many times.

One of the main limitations in developing 3D chip layouts currently lies in the performance of conventional coolers. More complex designs with extremely thin, stacked processors can reach power densities of up to 5 kW/cm3 (kilowatts per cubic centimeter)—a power density which exceeds that of any current heat engine, such as internal combustion engines, by ten times.

At IBM Research – Zurich, novel concepts to scale cooling technologies for 3D chip stacks are being explored. In test systems, water is piped directly between the individual chip layers through microscopic channels measuring only about 50 microns. Such designs allow 3D stacks of heating elements to be cooled very efficiently with the heat fluxes released by today's processors.

Before first fully functional prototypes can be realized, which is expected to happen in the next seven to ten years, researchers must still overcome several technical hurdles. Their aim is to develop a system with an optimized flow of water through the thin layers, which at the same time reliably isolates the electronics from the water. A special difficulty is posed by the thousands of electronic connections that run vertically through the chip stack. In fact, the density of the components in such a system would be comparable to that of the human brain, which is intersected by millions of nerve fibers for signal processing, and features tens of thousands of blood capillaries for nutrients and heat transfer—without interfering with each other.

The three-dimensional integration of computer chips is one of the most promising approaches to boost performance tremendously while reducing energy consumption considerably. Supercomputers as small as sugar cubes could thus become reality.

Provided by IBM