> As is typical in setting up large GPU clusters, we found that about 10% of the machines failed to boot, mostly due to physical issues with the servers. Some issues we encountered included: unconnected or miswired Ethernet cables, hardware issues in iDRAC, broken power supply units, bad NVME (nonvolatile memory express) drives, missing internal wires, and network cards or GPUs failing to show up.
This is a crazy high failure rate. Is this standard for traditional data centers too?
I did some of the work in the post (though mostly post-setup).
Speaking in generalities: the initial failure rates of these units are much higher than those of traditional non-GPU machines.
In general, the failure rates decline significantly during the operating life of hardware. So you deal with a bunch of issues up front that you try to resolve to reach a much more stable state.
I have also heard that failure rates on new GPUs are very high (approaching 20% if not burnt in), so that's unsurprising.
It's the other stuff I was more surprised about. I would have guessed that having your Ethernet cables plugged in and power supplies tested was table stakes nowadays. Then again, I've never been a datacenter admin...
3 comments
[ 3.7 ms ] story [ 15.3 ms ] thread> Step 2: Diagnosing broken machines
> As is typical in setting up large GPU clusters, we found that about 10% of the machines failed to boot, mostly due to physical issues with the servers. Some issues we encountered included: unconnected or miswired Ethernet cables, hardware issues in iDRAC, broken power supply units, bad NVME (nonvolatile memory express) drives, missing internal wires, and network cards or GPUs failing to show up.
This is a crazy high failure rate. Is this standard for traditional data centers too?
Speaking in generalities: the initial failure rates of these units are much higher than those of traditional non-GPU machines.
In general, the failure rates decline significantly during the operating life of hardware. So you deal with a bunch of issues up front that you try to resolve to reach a much more stable state.
There was a recent Meta engineering blog post that echoed some of our own experiences wrangling GPUs and high performance networks: https://engineering.fb.com/2024/06/12/data-infrastructure/tr...
It's the other stuff I was more surprised about. I would have guessed that having your Ethernet cables plugged in and power supplies tested was table stakes nowadays. Then again, I've never been a datacenter admin...