TeraPipe Token-Level Pipeline Parallelism for Training Large-Scale Language Models

TeraPipe:Token-LevelPipelineParallelismforTrainingLarge-ScaleLanguageModelsZhuohanLi1SiyuanZhuang1ShiyuanGuo1DanyangZhuo2HaoZhang1DawnSong1IonStoica1Abstractbitfloating-pointnumbers.Thissignificant...