Abstract
Transformer-based models are unable to process long sequences due to theirself-attention operation, which scales quadratically with the sequence length.To address this limitation, we introduce the Longformer with an attentionmechanism that scales linearly with sequence length, making it easy to processdocuments of thousands of tokens or longer. Longformer's attention mechanism isa drop-in replacement for the standard self-attention and combines a localwindowed attention with a task motivated global attention. Following prior workon long-sequence transformers, we evaluate Longformer on character-levellanguage modeling and achieve state-of-the-art results on text8 and enwik8. Incontrast to most prior work, we also pretrain Longformer and finetune it on avariety of downstream tasks. Our pretrained Longformer consistently outperformsRoBERTa on long document tasks and sets new state-of-the-art results on WikiHopand TriviaQA.