Unfortunately a reality vague decision of the technique. They say you can train multiple models at once but wouldn't this require N times more memory? They also say that models that require heavy compute won't parallelize well which also seems just a bit absurd because aren't all those shaders all firing off perfectly in parallel to do all the big matrix math?
The problem to solve is GPU utilization rate. Training a “huge model” on n GPUs will have a cyclical pattern of idle GPUs and GPUs are expensive. This approach is likely scheduling GPU access across t training jobs to insure GPU is always in use. Maximizing GPU utility is a key requirement for training as a service.
The biggest issue is that the failure mode of this approach (one fault hoses the entire pipeline of T jobs) is exactly opposing the goals of its most likely users: Training SaaS providers.
5 comments
[ 0.24 ms ] story [ 22.7 ms ] threadhttps://ieeexplore.ieee.org/document/10130297
https://www.computer.org/csdl/journal/tc/5555/01/10130297/1N...
The paper has a better way to share one GPU between small training jobs.
The real world problem is training huge models for a long time across many GPUs.
The problem the paper solves is one that is practically irrelevant aside from some niche cases.
The biggest issue is that the failure mode of this approach (one fault hoses the entire pipeline of T jobs) is exactly opposing the goals of its most likely users: Training SaaS providers.