Zero-Offload: Democratizing Billion-Scale Model Training (arxiv.org)

> ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone. It can train models with over 13 billion parameters on a single GPU, a 10x increase in size compared to popular framework such as PyTorch, and it does so without requiring any model change from the data scientists or sacrificing computational efficiency.

Official open-source implementation: https://www.deepspeed.ai/