Low-Rank Bottleneck in Multi-head Attention Models

Low-RankBottleneckinMulti-headAttentionModelsSrinadhBhojanapalli1ChulheeYun2AnkitSinghRawat1SashankReddi1SanjivKumar1Abstracttotherecurrentmodels.Selfattentionmodelsalsohavefoundapplicationsinvisio...