28 comments

[ 4.9 ms ] story [ 71.4 ms ] thread
> TPU chips are hooked up in series in the loop, which naturally means some chips will get hotter liquid that has already passed other chips in the loop. Cooling capacity is budgeted based on the requirements of the last chip in each loop.

Of course, it's worth noting that if you've got four chips, each putting out 250W of power, and a pump pushing 1 litres of water per minute through them, water at the outlet must be 14°C hotter than water at the inlet, because of the specific heat capacity of water. That's true whether the water flows through the chips in series, or in parallel.

The equations all work out OK though since one can go substantially above a litre a minute at pressures which are still viable. The standard 18W thing for desktops is about ten times that.
It’s very odd when mainframes (S/3x0, Cray, yadda yadda) have been extensively water-cooled for over 50 years, and super-dense HPC data centers have used liquid cooling for at least 20, to hear Google-scale data center design compared to PC hobbyist rigs. Selective amnesia + laughably off-target point of comparison.
> super-dense HPC data centers have used liquid cooling for at least 20

Hasn't this just been for things like rack doors and such?

In the last ~two generations of servers it seems like now there's finally DLC (direct liquid cooling) into the actual servers themselves (similar to the article). Intel kind of forced that one on us, with their high-end SKUs. This has been a pain becuase it doesn't fit into legacy datacenters as easily as the door/rack-based systems.

I won't say which server vendor it is, but I've put in more than one service ticket for leaking coolant bags (they hang them on the racks).

I wonder what the economics of water cooling really is.

Is it because chips are getting more expensive, so it is more economical to run them faster by liquid cooling them?

Or is it data center footprint is more expensive, so denser liquid cooling makes more sense?

Or is it that wiring distances (1ft = 1nanosecond) make dense computing faster and more efficient?

> Or is it that wiring distances (1ft = 1nanosecond) make dense computing faster and more efficient?

Contrary to other posters, I'd argue this effect is relatively small. A really good interconnect fabric might give you ping-pong times on the order of 1 microsecond, which is still 1000 times larger than a nanosecond. Most of the delay will be in the switches and the end nodes, not in the signal traveling over the wire or fiber. Say for a large-ish cluster with a diameter of, say, 100 feet (something like 7 rows of racks, each row 100 feet long, give or take), if liquid cooling allows you to double the density, you could condense it to a diameter of 100/sqrt(2) = 70 ft (about 5 rows of 70 ft each). As a ping-pong involves a signal going both ways, the worst-case increase in signal delay would be (100-70)*2 = 60 ft or 60 nanoseconds (in reality somewhat more since cables have to be routed). So about a 6% increase if we assume the baseline is 1 microsecond. Measurable, yes, but likely very small effect on application performance vs. a ping-pong microbenchmark.

Now where it can matter is that by packing the components more closely together, you can connect more chips via backplane and/or copper connectors vs. having to use optics.

It's also the fact that a large portion of the power used by a data centre is actually spent on cooling, so anything that will make that more efficient is a cost saving.
>"Google extensively validates components with leak testing, uses alerting systems to discover problems like leaks, and takes preventative measures like scheduled maintenance and filtration. They also have a clear set of protocols to respond to alerts and issues, allowing a large workforce to respond to problems in a consistent fashion. It’s a far cry from the ad-hoc measures enthusiasts take to maintain their water cooling setups."

How about the author compare it to MS, Meta, or AWS instead of Joe Blow buying parts online? I would hope that Google had extensive validation and clear protocols. [Roll-Eyes]

How much moat Google has in the AI race due to its in-house expertise in data centers and in-house IPR in TPUs? Can any company on Earth match them?
I once saw an interview with the SVP who oversees Azure datacenter buildout or something like that and a thing that stuck with me was that he said his job got a lot easier when he realized he was no longer in the computer business, he was now in the industrial cooling business.

Reading this article immediately made me think back to that.

curious to see that interview, do you remember link?
What I don't understand is why not use what I've seen in industrial processes? Why not a low boiling point liquid and a heat exchanger? Or even better power a turbine like biogas plants do
Between 2006 and 2012 I was in DCs. I didn’t live there but I had to visit at stupid o’clock too often.

It’s not a hospitable place.

I would have loved it if cooling was quieter and less extreme.

The reason interfaces are on the back is that’s the intake side. So, bring a sweater.

Or just go to the other side of the rack to warm your hands back up.

In theory data center cooling is simple - CPUs run at 60-70C, while the outside ambient is usually 30C or less, so heat should just "flow downhill" with a bit of help from fans, pumps, etc.

The problem with using air cooling to get it there is that the humans who run the data center have to enter and breathe the same air that's used to cool the computers, and if your working fluid gets too hot it's quite unhealthy for them. (we run our hot aisles at 100F, which is a bit toasty, and every third rack is a heat exchanger running off the chilled water lines from the outside evaporative cooler, modulo a heat exchanger to keep the bird shit out)

We're not going to be able to pump much heat into the outside world unless our working fluid is a decent amount hotter than ambient, so when it gets reasonably warm outside we need to put chillers (water-to-water AC units) in the loop, which consume energy to basically concentrate that heat into a higher-temperature exhaust. When it's really hot outside they consume quite a bit of energy.

If the entire data center was liquid cooled we could have coolant coming from the racks at a much higher temperature, and we'd be able to dump the heat outside on the hottest days without needing any chillers in the loop. As it is we have some liquid cooled racks running off the same chilled water lines as the in-aisle heat exchangers, but the coolant temp is limited by the temperature of our hot aisles, which are quite hot enough already, thank you.

I wonder if we will eventually have people entering datacenters in heatsuits to eke out some extra efficiency gains from just needing to cool people.
> CPUs run at 60-70C, while the outside ambient is usually 30C or less, so heat should just "flow downhill" with a bit of help from fans, pumps, etc.

This isn't how you think of heat flow. The CPUs run at a given power. Their temperature will depend on the ambient temperature, and on the thermal impedance between them and the ambient. If the thermal impedance is too high, they'll be too hot and die.

> CPUs run at 60-70C

When properly cooled. If cooling is insufficient they can easily get much hotter.

OVH has been doing liquid cooling for quite a while, and is now using immersion also. Somewhat surprising to see at that dirt cheap end of the market.
I wish I could be as excited as I used to be about Google's infrastructure. But with their constant attacks on the freedom of the internet, they have exhausted my good-will.

So I am afraid I do not care about your liquid cooling on this day. I'm also not particularly impressed, I am sure the devil is in the details at this scale but there doesn't seem to be much novel here.

If a Googler happens to read this and it makes them sad, it's Google, not you. You might consider doing cool stuff somewhere else.

FAANGs have gone from wunderkind darlings that would pave the way for progress, like turn of the 20th-century-style, to monopolistic, feudal overlords rapaciously seeking to exploit land and resources with zero concern for the environment or locals' quality of life.
> In theory, a PC builder could try exchanging heat to a toilet tank, allowing for highly efficient cooling with each flush.

The future is here.

I’ve always wondered: what if we could actually use this heat? For example, could we take the heated water from cooling a data center and pump it to places that need it (Greenhouses, Industrial processes, Desalination and water treatment)?
That's pretty much the way it had to go, other than doing full immersion cooling like GRC that never caught on. Meta has various rack-based chill water heat-exchanger cooling solutions but this looks a bit more integrated.