SCROLLS: Standardized CompaRison Over Long Language Sequences

  • 2022-01-10 18:47:15
  • Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy
  • 51

Abstract

NLP benchmarks have largely focused on short texts, such as sentences andparagraphs, even though long texts comprise a considerable amount of naturallanguage in the wild. We introduce SCROLLS, a suite of tasks that requirereasoning over long texts. We examine existing long-text datasets, and handpickones where the text is naturally long, while prioritizing tasks that involvesynthesizing information across the input. SCROLLS contains summarization,question answering, and natural language inference tasks, covering multipledomains, including literature, science, business, and entertainment. Initialbaselines, including Longformer Encoder-Decoder, indicate that there is ampleroom for improvement on SCROLLS. We make all datasets available in a unifiedtext-to-text format and host a live leaderboard to facilitate research on modelarchitecture and pretraining methods.

 

Quick Read (beta)

loading the full paper ...