Intel does math on oil-dunk test for cooler servers

Sep 03, 2012 by Nancy Owano report
Supermicro SuperServers submerged at CGGVeritas

(Phys.org)—Intel just finished a yearlong test of Green Revolution Cooling's mineral-oil server-immersion technology. Intel has tried immersing servers in the company's oil formulation to keep the servers cool and they report good results. The Intel servers were subjected to a yearlong bath in boxes filled with the oil-based coolant. Intel's results are one more way for the Austin, Texas-based company to pass along a convincing message that dipping a data center's servers in oil is not crazy but a sane way to cool the data center's power-hungry machines. The company believes its liquid cooling enclosures can cool high-density server installations at less cost. "We can reduce cooling energy use by 90 to 95 percent while also reducing server power by 10 to 20 percent," according to the company.

Intel reports that the servers ran at a PUE just above 1.0, and showed no damage. Translation: Traditional air-cooled server racks often operate at a Power Usage Effectiveness rating of about 1.6. Intel's oil-immersed servers operated at a PUE between 1.02 and 1.03, according to Mike Patterson, senior power and thermal architect at Intel.

Patterson said the oil-dunking approach is now in Intel's evaluation phase. Intel will need to understand the fuller implications of oil-optimized platforms.

Potentially, the approach may reduce the cost of running data centers, as cooling through the oil immersion system can bring reductions in average and peak . Those taking their first leap to use this oil immersion approach would be those who want as much power as possible coming out of their computing operations with as little as possible applied toward cooling.

This video is not supported by your browser at this time.

Patterson said that the company found the technology to be safe for its server components. At the end of the test period, the Intel servers were placed in its failure-analysis lab, which did not find any damaging effects.

Green Revolution Cooling's product is called the CarnotJet System, defined as a fluid submersion cooling solution for data center servers. It uses nonconductive liquid coolant rather than air to achieve maximum cooling efficiency and performance.

Mention "oil" and add "servers immersed in oil" and for those not familiar with the results, this can sound very strange if not risky, Intel's Patterson said. That initial reaction is worth getting past. Patterson said Intel's research into how to build oil-optimized servers could result in a reference architecture around which server manufacturers could begin building such systems.

Already, Green Revolution Cooling has a list of customer centers running computer systems that make use of the approach. These include the Oil/Gas Seismic Processing Center; European Colocation Center; East Coast Corporate Data Center; US Research Institute; and the Texas Advanced Computing Center.

The coolant is clear, has a light viscosity, is odorless and is nontoxic. The company calls its coolant GreenDEF, which is a formulation of nonconductive white . Their formulation is optimized for cooling servers and other data center hardware.

When compared to present-day design principles for keeping servers cool, the oil-immersion approach could have far-reaching effects. That is what Patterson means by its having an impact on a reference architecture around which server manufacturers can build their systems.

Oil immersion would do away with the need for raised floors and other requirements for air cooling. According to , energy is saved by removing server and power supply fans from the servers, as they are no longer necessary when servers are submerged.

The company also touts the advantage of server reliability, as the do not collect dust. Dust is a thermal insulator that raises server temperatures and reduces reliability. One recent study, they said, showed that dust accumulation increased power draw by 2 percent due to server temperature increases. The circulation of the coolant is said to be "excellent" with uniform temperatures without hot spots.

Explore further: Taking great ideas from the lab to the fab

More information:
gigaom.com/cloud/intel-immerse… il-and-they-like-it/
www.grcooling.com/

Related Stories

HP picks Intel's Centerton for low-power server rollout

Jun 22, 2012

(Phys.org) -- Hewlett-Packard has a longer term message that reaches over all company announcements. HP wants the world to know that, never mind hasty CEO exits and entrances, never mind killing off brand-conscious ...

IBM Rolls Out Low-Power Servers

Apr 12, 2007

In addition to offering a number of low-watt AMD and Intel processors, Big Blue will also offer a flash drive in one its blade systems as a way to reduce power consumption.

Doubling Down On Intel-Based Servers

Jul 11, 2005

Intel Corporation today introduced the dual-core Intel Pentium D processor-based platform for entry-level servers that includes two processor cores in one processor and the corresponding Intel E7230 chipset.

IBM Unveils New Dual-Core X3 Servers

Nov 01, 2005

IBM is upgrading its X3 Architecture-based family of servers with the introduction today of new, dual-core systems based on the dual-core Intel Xeon processor 7000 sequence, formerly code named "Paxville MP." X3 Architecture-based ...

Recommended for you

Law changed to allow 'unlocking' cellphones

11 minutes ago

President Barack Obama signed a bill into law on Friday making it legal once again to unlock a cellphone without permission from a wireless provider, so long as the service contract has expired.

Social network challenges end in tragedy

13 minutes ago

Online challenges daring people to set themselves ablaze or douse themselves in ice water are racking up casualties and fueling wonder regarding idiocy in the Internet age.

Microsoft sues Samsung alleging contract breach

13 minutes ago

Microsoft on Friday sued Samsung in federal court claiming the South Korean giant had breached a contract over cross-license technology used in the fiercely competitive smartphone market.

States debate digital currency

1 hour ago

Now that consumers can use digital currencies like bitcoin to buy rugs from Overstock.com, pay for Peruvian pork sandwiches from a food truck in Washington, D.C., and even make donations to political action committees, states ...

User comments : 23

Adjust slider to filter visible comments by rank

Display comments: newest first

M_N
1.7 / 5 (6) Sep 03, 2012
As an electronic engineer, I wonder how the oil affects non hermetically sealed components like electrolytic capacitors? I guess they've considered that, but I'd like to see a study longer than 1 year before I dunked hundreds of millions of $ worth of hardware in oil...
Shootist
1 / 5 (3) Sep 03, 2012
high quality motherboards and peripherals use solid caps.

CrashM
not rated yet Sep 03, 2012
It would require specifically designed servers to an extent... a fan/heatsink designed for air would not be very efficient at circulating oil... you would need pumps to circulate the oil and possibly heat sinks with wider fins to allow better oil movement.
Sanescience
2 / 5 (4) Sep 03, 2012
It also might be possible to monitor the oil for contaminants to detect unintended dissolving or chemical reactions.
axemaster
4 / 5 (2) Sep 03, 2012
It must make a hell of a mess if you have to remove or service anything...
ab3a
not rated yet Sep 03, 2012
It must make a hell of a mess if you have to remove or service anything...

Not really. Think of all the dust bunnies that a typical air-cooled system drags across a circuit board. I'm not sure which would be worse, but neither of them are desirable.

That said, this idea is not even remotely new. Overclockers have been doing this for at least the last ten years or more. And mineral oil has been used as a coolant for power transformers since the beginning of electrical distribution.
kochevnik
1 / 5 (3) Sep 03, 2012
Not really. Think of all the dust bunnies that a typical air-cooled system drags across a circuit board.
Colocation HVAC filters are supposed to remove the dust. Though they in turn blow zinc particles around which short circuit computer boards. That's probably a greater danger.

No mention in the article how hard drives fare being immersed in oil.
LED Guy
5 / 5 (1) Sep 03, 2012
Cray used fluorocarbon immersion for their supercomputers back in the 1980s. Same concept here (where do you think they got the idea), they are just using a more environmentally friendly fluid.

Design your components properly and you could use water. Thin film boiling regime allows unbelievable heat transfer coefficients. Before everyone says thats too hot, consider that water can boil at (or below) room temperature if you reduce the pressure.
rwinners
1 / 5 (2) Sep 03, 2012
But is it 'green' oil?????

And guys?? Intel has tested this system. Why do you doubt it?
DonaldJLucas
5 / 5 (3) Sep 04, 2012
I am involved in cooling printed circuit boards in oil. There are several materials that have to be avoided for the system to be reliable, but if you avoid them, a system can last much longer under oil than when cooled by air or other liquids. The primary materials that have to be avoided are soft rubbery materials and polymers that, over time, will dissolve in oil. Silicone rubber can be one of these, depending on which oil is being used. So far, testing has not found any problems with the epoxy used to overmold integrated circuits. Circuit boards must be made of special FR4 prepregs that are a little more expensive than the standard pcb materials or the pcb may delaminate over time. In particular, PVC, a common wire insulation material, may become brittle over time and stop being as good an insulator, but these are problems that won't manifest for at least 5 years after the oil is applied. With proper material selection, 30 year life is possible.
DonaldJLucas
5 / 5 (2) Sep 04, 2012
I am surprised to find that they are using mineral oil. There are many types of mineral oil and I am not an expert in this particular area, but most transformer companies and utilities are trying to switch over to synthethic oils made primarily from corn oil. The new synthethic esters are environmentally friendly and non-toxic. One reason that mineral oils are to be avoided is because they have a relatively low flash point, around 130 degrees C for common transformer oil. The new synthethics don't flash below 300C and can be used indoors without any special considerations whereas transformers in mineral oil must be located outside a building, or in a fire-proof room because of the low flash point of mineral oil.
antialias_physorg
1 / 5 (1) Sep 04, 2012
It must make a hell of a mess if you have to remove or service anything...

Since there's not much to service on a motherboard that's not much of an issue.

but most transformer companies and utilities are trying to switch over to synthethic oils made primarily from corn oil.

Transformers for power applications require a lot more heat to be disspated than indivdiual circuit boards. Using synthetic oils for the application mentioned would be overkill - and very expensive (most processors aren't able to run beyond 70C for too long without taking damage).

But is it 'green' oil?

Since it's not burned: yes.

No mention in the article how hard drives fare being immersed in oil.

Because they are not immersing hard drives in oil.
Why would they? Harddrives don't overheat. (And even if: harddrives are vacuum sealed...so there's really no point in immersing them in oil in the first place)
Eikka
3.7 / 5 (3) Sep 04, 2012
No mention in the article how hard drives fare being immersed in oil.


They don't. Hard drives are not sealed because they rely on air pressure to float the magnetic head. They are NOT vacuum sealed, but simply filtered, and the centrifugal action keeps any dust off the platters.

Harddrives don't overheat.


Yes they do. Ever touched a hard drive after a backup-session? It's hot, and the disc bearings and electronics don't like it.

Since there's not much to service on a motherboard that's not much of an issue.


You do need to handle the connectors and cabling, and accessing a single server blade for troubleshooting or testing it in case of failure is more difficult, as well as replacing one because you need to pull it out and drain the oil.

Perhaps the best practice is to use a plastic bucket to move the parts around, or design the server to simply disconnect broken units and swap them out all at once when convenient.
ECOnservative
1 / 5 (3) Sep 04, 2012
What are the effects of component failure (say, a blown capacitor or some other 'smoke release' event) on the cooling solution? Does it need to be constantly circulated and filtered? What are the heat-exchanger designs?
krundoloss
1 / 5 (2) Sep 04, 2012
This sounds neat, but hardly practical for anything but the largest of datacenters. I would not like it, it would add complexity and mess to maintenance of servers. I do like that the components are made well enough and sealed so that they function while submerged!
antialias_physorg
not rated yet Sep 04, 2012
This sounds neat, but hardly practical for anything but the largest of datacenters.

You can buy mineral oil cooled/submerged DIY kits on the internet for your home PC.
This site talks about their experience with it:
http://www.pugets...rged.php

(and I gather they have been running the submerged PC for a number of years 24/7)

Excerpt:

Q:Won't the mineral oil eat away at the rubber, making capacitors blow or components fall apart?

A: Some people say the motherboard will fall apart, others that the acrylic tank will dissolve away to nothing! In reality, we have seen absolutely zero effect. All components are perfectly intact, and the system remains rock solid. The only impact we have seen is on adhesives -- the label stickers on the memory came off
...
But to answer the cries of doom and gloom -- we've seen no indications yet. All the rubber seals are intact, and the capacitors are completely unaffected
DonaldJLucas
not rated yet Sep 04, 2012
I don't work for Midel, a manufacturer of a premium synthethic ester oil (7131) suitable for this type of product, but you can read more about it at http://www.midel.com/. There is a list of known incompatibilities with this particular oil (probably quite similar to other mineral oils) at http://www.midel....ibility. Our own program performs chemical analysis on the oil after testing all construction materials for 3 months in oil at elevated temperatures. This can predict future failures when prorated properly. In the field, you can sample the oil once a year and a chemical analysis will show if any components in the tank have experienced "smoke" events or if any compounds are starting to break down. Incidentally, while the full synthetics started out very expensive (just like synthetic motor oils), the price is now at only a small premium over transformer grade mineral oil and the synthetic is formulated far more consistently than mineral oil.
DonaldJLucas
not rated yet Sep 04, 2012
Note: Delete the period at the end of the link in my previous message. Not sure why the link captured the period at the end of the sentence.
Eikka
1 / 5 (3) Sep 04, 2012
There's one annoying property of oil-submerged parts: capillary action pulls oil along all the cables, on the inside of the cable. For example, a network cable will happily siphon the tank empty unless you poke holes in the jacket.
rwinners
1 / 5 (2) Sep 04, 2012
Oil (and other liquids) does a much better job of transferring heat than air does. I'm sure that the oil is filtered and cooled as it circulates.
Sanescience
1 / 5 (3) Sep 04, 2012
I'm guessing that they don't want to use an alcohol because of it's flammable nature and the need to collect and re-condense the vapor.

Though it does sustain impressively cold temperatures, freezing a little below dry ice ( perhaps supporting some performance edge over competition ).

Would also be allot easier to service the equipment, just pull out of the pool, put a fan on it, and let it warm up in case of cryo cooling. Will be perfectly clean and dry in a minute or so.

As for hard drives, solid state drives are taking over, and when performance counts are preferred.
antialias_physorg
not rated yet Sep 04, 2012
Would also be allot easier to service the equipment, just pull out of the pool, put a fan on it

However anyone working in that environment would have to wear full breathing gear. Alcohol is also a lot more expensive than minearal oil.

There's one annoying property of oil-submerged parts: capillary action pulls oil along all the cables

Capillary action doesn't allow the substance to climb infinitely high (see the picture). It also depends on the surface tension of the substance. With a very simple surfactant that surface tension can be reduced (but for oil that's hardly necessary)
chromosome2
1 / 5 (1) Sep 05, 2012
But is it 'green' oil?????

And guys?? Intel has tested this system. Why do you doubt it?


Like they tested their under-IHS thermal interface material for Ivy Bridge too.. "Oh, well, the servers will last two years... you really should be buying new ones by then anyways.."