5 comments

[ 2.6 ms ] story [ 232 ms ] thread
Super interesting, thanks for putting this together
Interesting. This is probably a universal problem for large model training but not being discussed enough.
Super interesting problem that's affecting more people than they probably realize.
Interesting post. It would be much better if the author included a few code snippets to show how to identify the failing GPU during training.