Server immersion cooling
Server immersion cooling is a computer cooling practice by which computer components such as CPUs, GPUs, SSD and DDR Memory, and other electronics including complete servers are completely submerged in a thermally conductive dielectric liquid or coolant, which is then cooled through the use of a circulation system employing liquid pumps, plumbing, liquid-to-liquid heat exchangers and/or dry cooler (radiator) type coolers to reject the heat from the coolant. Common dielectrics coolants which are used in liquid immersion cooling of electronics are either mineral-oil based or entirely synthetic chemicals. These chemicals fall into two broad groups of application use cases: Single-phase, Liquid Immersion Coolants (SLIC) or 2-phase, Liquid Immersion Coolants (2PIC).
Server immersion cooling has the potential of becoming a popular server cooling solution, in particular for green data centers, as it allows operators to drastically reduce their energy usage through the elimination of the expensive air cooling infrastructure including on-board server fans, CRACs, A/C compressors, air-circulation fans, necessary duct work, air handlers, and other active ancillary systems such as dehumidifiers. These systems are replaced with efficient low speed liquid circulation pumps and heat exchanger and/or dry cooler systems. The power reduction that can be achieved through the use of liquid immersion cooling is often measured in terms of Power Usage Effectiveness (PUE). However, It is important to note that the definitions of PUE for air-cooled devices and liquid immersion cooled devices differs in one important aspect, which makes such direct comparisons misleading in favor of air cooling.
- PUE for air-cooled data centers, as defined by the Green Grid, includes the power used by the fans and other active cooling components found on-board the servers as part of the IT Equipment Energy,
- PUE for liquid immersion cooling excludes these values from the IT Equipment Energy component because these system elements (in particular on board fans) are generally removed from the IT equipment as they are not necessary to circulate the dielectric coolants, and often actually impede the circulation of the dielectric coolant.
This discrepancy in the definition of PUE for the different cooling methods results in the PUE of air-cooled data centers generally being overstated by anywhere from 10-20% when compared against the PUE of a liquid immersion cooled facility of the same power usage.
Servers and other IT hardware cooled by liquid immersion cooling do not require fans to circulate the dielectric coolants, thus they are removed from the system prior to immersion. Hydrocarbon based thermal pastes typically used on heat spreaders for CPUs and other chips needs to be replaced with a different compound in order to avoid the thermal paste from dissolving in the coolant. Indium foil and thermally conductive epoxies are commonly used as a replacement materials.
The temperatures used in liquid immersion cooling are determined by the highest temperature at which the devices being immersed can reliably operate. For servers this temperature range is typically between 15 to 65 °C (59 to 149 °F), however in ASIC based crypto mining devices this range is often extended up to 75C. This increase in the high end of the temperature range allows data centers operators to use entirely passive dry coolers, or much more efficient evaporative or adiabatic cooling towers instead of chiller-based air cooling or water chillers. This increase in the temperature range also allows operators using Single-phase, Liquid Immersion Coolants to more effectively use the change in outdoor temperatures to get more efficient cooling from their systems because the single-phase systems are not limited in their effectiveness by the boiling point of the coolant as is the case with 2-phase coolants.
Current commercial applications for immersion cooling range from data center oriented solutions for commodity server cooling, server clusters, HPCC applications and Bitcoin Mining. and mainstream cloud-based and web hosting architectures. Electric vehicle and battery manufacturers also employ liquid immersion cooling in batteries, drive-train, Kinetic energy recovery systems, electric motors, electric motor controllers, and other on-board electronic sub systems. Liquid immersion cooling is also used in the thermal management of LEDs, Lasers, X-Ray machines, and Magnetic Resonance Imaging devices.
The use of mineral or vegetable-based dielectrics liquid for cooling large power distribution components such as transformers and industrial machinery is a common practice and has been successfully deployed since the beginning of the 1900s
- Immersing electronics (specifically transformers) in dielectric fluids for thermal management was used before 1887.
- The first patent to explicitly mention the use of oil as a coolant and insulator is in the patent submitted for a Constant Current Transformer in 1899 by Richard Fleming of Lynn, Massachusetts, assignor to the General Electric Company of New York
- The first reference to the specific use of dielectric fluids being used to cool "computers" is in 1966 by Oktay Sevgin of IBM.
- In 1968, Richard C Chu and John H Seely, working for IBM patented an "Immersion cooling system for modularly packaged components."
- Seymour R. Cray, Jr. founder of Cray Research, LLC patented a "Immersion cooled high density electronic assembly" in 1982.
- The Cray T90 (released in 1995) used large liquid-to-chilled liquid heat exchangers and single or two-phase immersion cooling oils for heat removal
A different industry where immersion cooling is applied to electronic components is deep sea research where Remotely Operated Underwater Vehicles with electronic equipment are filled with newtonian Single-phase, Liquid Dielectrics to both protect them from corrosion in seawater and as a pressure compensating fluid to avoid being crushed by the extreme pressure exerted on the ROV while working in the deep sea. This application also includes the cooling of the electric motors used for under sea propulsion.
Until about 2014, the technology was typically only utilized in special very intensive supercomputing projects, similar to the Cray Computer Applications mentioned above. However, since the dramatic increase in global energy consumption by data centers, there is now a focus on energy efficiency which has driven the utilizing of liquid immersion cooling in both data centers and crypto mining operations to reevaluate its application. In addition the advent of new very high density CPUs and GPUs for use in real-time processing, Artificial intelligence, machine learning, and data mining operations is pressuring users and data center operators to evaluate liquid immersion cooling for ability to cool high density racks as well as reduce the total mechanical footprint of data centers.
Since 2005, several companies tried to claim patents based on the open bath approach with external pump installations. Most, if not all of these claims have been denied on the basis that IBM had already patented the technology.
Server Immersion Cooling techniques
Open or Semi-Open Bath Immersion Cooling
Open or semi-open bath immersion cooling is a data center cooling technique that implies fully submerging servers in dielectric coolant fluid. It is called open or semi-open bath because servers are placed side-by-side in large tanks that function as baths. These baths operate at atmospheric pressure and allow the coolant fluid to be moved through the hardware components or servers submerged in it.
Open Bath Immersion Cooling uses single-phase dielectric liquids. These liquids always remain in liquid state while operating. They never boil or freeze. The coolant is usually pumped to an external heat exchanger where it is cooled either through the use of a liquid-to-liquid heat exchanger or a dry cooler (radiator). When the heat density is quite low natural convection can also be used to circulate a dielectric oil through the servers. and a water cooled heat exchanger. This technique uses “open baths“ as there’s little (or no) risk of Single-phase, Liquid Immersion Coolants evaporating.
In semi-open bath Immersion Cooling, a two-phase chemically engineered liquid is the heat transfer fluid. Heat is removed in a 2-phase system through the phase change the coolant undergoes at its operating temperature, it literally boils due to its low boiling point. The system takes advantage of a concept known as “latent heat” which is the heat (thermal energy) required to change the phase of a fluid, this occurs when the 2-phase coolant comes in contact with the heated electronics in the bath that are above the coolants boiling point. Once the 2-phase coolant enters its gas phase it must be cooled or condensed, typically through the use of water cooled coils placed in the top of the tank. Once condensed the 2-phase coolant drips back into the primary cooling tank. The 2-phase coolant in the tank generally remains at it “saturation temperature”. Energy transferred from the servers into the two-phase coolant will cause a portion of it to boil off into a gas. The gas rises above the liquid level where it contacts a condenser which is cooler than the saturation temperature. This causes the gaseous state coolant to condense back into a liquid form and fall back into the bath.
The term "semi-open bath" means, that when the system operates, it is sealed to avoid the evaporation of the coolant.
Sealed Server Immersion Cooling
A different approach to open or semi-open bath immersion cooling are sealed server immersion cooling solutions, where servers are specially built in a liquid-tight casing. The dielectric coolant is circulated inside or pumped through each server to collect heat from the components. The heat is then taken to a manifold or backplane in the rack where it is either circulated directly outside the building to a cooling tower or heat exchanger or cooled directly at the rack with a chilled water circuit.
The main advantage of this approach is that servers are mounted in self contained vessels that can be easily replaced in the rack and are not required to be placed in horizontal baths. The most common disadvantage is that not all hardware can be used as the vendor defines the hardware specs of the sealed servers.
- "Eight emerging data center trends to follow in 2016". Robert Gates. December 15, 2015.
- "Indium Foil - a Thermal Interface Material". Retrieved 2017-07-25.
- "Data center liquid immersion cooling with adiabatic cooling towers". Submer Technologies. January 2, 2016.
- "Ice X: Intel and SGI test full-immersion cooling for servers". Computerworld, Inc. April 8, 2014.
- "Facebook throws servers on their back in HOT TUBS of OIL". The Register. October 14, 2013.
- "Liquid immersion cooling relief for ultra-dense data centers". TechTarget. October 5, 2014.
- "What is liquid immersion cooling? - Definition from WhatIs.com". WhatIs.com. Retrieved 2017-07-25.
- "Immersion Cooling Steps Up for HPC Clusters". insideHPC. May 7, 2014.
- "BitFury to Launch Energy Efficient Immersion Cooling Data Center". Business Wire. December 11, 2015.
- "The original transformers were replaced with Stanley oil filled transformers in 1904". The Folsom Power Plant 1895. Edison Tech Center.
- "Transformer's History and its Insulating Oil" (PDF).
- "Constant Current Transformer" (PDF).
- "Multi-liquid heat transfer".
- "Immersion cooling system for modularly packaged components".
- "Immersion cooled high density electronic assembly".
- "Fluid Selection and Property Effects in Single and Two-Phase Immersion Cooling" (PDF). JOHN R. SAYLOR, AVRAM BAR-COHEN, SENIOR MEMBER, IEEE, TIEN-YU LEE, TERRY W. SIMON, WE1 TONG, AND PEY-SHEY WU. November 4, 1988.
- "Electronics Take a Bath" (PDF). Lawrence Berkeley National Laboratory. November 5, 2014.
- "Asperitas Immersed Computing". www.asperitas.com. Rolf Brink. 2017-01-30. Retrieved 2017-07-24.
- "3M™ Novec™ 7100 Engineered Fluid". © 3M.
- "Immersion Cooling with 3M(TM) Novec(TM) Engineered Fluids". 3M. April 8, 2014.
- "Immersion-2 Rack Platform (PUE 1.01)". AlliedControl. January 22, 2014.
- "Targeted liquid cooling for a system". Rackspace. March 23, 2011.
- "Sealed Server Product Example". LiquidCool Solutions. January 1, 2015.