torchgpipe, a GPipe Implementation in PyTorch

https://github.com/kakaobrain/torchgpipe

Kakao Brain announces torchgpipe, an implementation of GPipe in PyTorch as a handy library.

GPipe is a scalable pipeline parallelism library published by Google Brain. It leverages the training of a giant model which requires much memory. For instance, Google trained AmoebaNet-B with 557M parameters over GPipe.