XNLI: Evaluating Cross-lingual Sentence Representations

  • 2018-09-13 16:39:53
  • Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, Veselin Stoyanov
  • 19

Abstract

State-of-the-art natural language processing systems rely on supervision inthe form of annotated data to learn competent models. These models aregenerally trained on data in a single language (usually English), and cannot bedirectly used beyond that language. Since collecting data in every language isnot realistic, there has been a growing interest in cross-lingual languageunderstanding (XLU) and low-resource cross-language transfer. In this work, weconstruct an evaluation set for XLU by extending the development and test setsof the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15languages, including low-resource languages such as Swahili and Urdu. We hopethat our dataset, dubbed XNLI, will catalyze research in cross-lingual sentenceunderstanding by providing an informative standard evaluation task. Inaddition, we provide several baselines for multilingual sentence understanding,including two based on machine translation systems, and two that use paralleldata to train aligned multilingual bag-of-words and LSTM encoders. We find thatXNLI represents a practical and challenging evaluation suite, and that directlytranslating the test data yields the best performance among availablebaselines.

 

Quick Read (beta)

loading the full paper ...