PipeTransformer Automated Elastic Pipelining for Distributed Training of Large-scale Models

PipeTransformer:AutomatedElasticPipeliningforDistributedTrainingofLarge-scaleModelsChaoyangHe1ShenLi2MahdiSoltanolkotabi1SalmanAvestimehr1AbstractTransformer(ViT)(Dosovitskiyetal.,2020)alsoachieved...