Silent Data Corruptions: The Boogeyman of LLM Training (adept.ai)

Fascinating read!

Super interesting, thanks for putting this together

Interesting. This is probably a universal problem for large model training but not being discussed enough.

Super interesting problem that's affecting more people than they probably realize.

Interesting post. It would be much better if the author included a few code snippets to show how to identify the failing GPU during training.