Abstract
Position encoding in transformer architecture provides supervision fordependency modeling between elements at different positions in the sequence. Weinvestigate various methods to encode positional information intransformer-based language models and propose a novel implementation namedRotary Position Embedding(RoPE). The proposed RoPE encodes absolute positionalinformation with rotation matrix and naturally incorporates explicit relativeposition dependency in self-attention formulation. Notably, RoPE comes withvaluable properties such as flexibility of being expand to any sequencelengths, decaying inter-token dependency with increasing relative distances,and capability of equipping the linear self-attention with relative positionencoding. As a result, the enhanced transformer with rotary position embedding,or RoFormer, achieves superior performance in tasks with long texts. We releasethe theoretical analysis along with some preliminary experiment results onChinese data. The undergoing experiment for English benchmark will soon beupdated.