Corpus Similarity Measures Remain Robust Across Diverse Languages

  • 2022-06-09 09:17:16
  • Haipeng Li, Jonathan Dunn
  • 7

Abstract

This paper experiments with frequency-based corpus similarity measures across39 languages using a register prediction task. The goal is to quantify (i) thedistance between different corpora from the same language and (ii) thehomogeneity of individual corpora. Both of these goals are essential formeasuring how well corpus-based linguistic analysis generalizes from onedataset to another. The problem is that previous work has focused onIndo-European languages, raising the question of whether these measures areable to provide robust generalizations across diverse languages. This paperuses a register prediction task to evaluate competing measures across 39languages: how well are they able to distinguish between corpora representingdifferent contexts of production? Each experiment compares three corpora from asingle language, with the same three digital registers shared across alllanguages: social media, web pages, and Wikipedia. Results show that measuresof corpus similarity retain their validity across different language families,writing systems, and types of morphology. Further, the measures remain robustwhen evaluated on out-of-domain corpora, when applied to low-resourcelanguages, and when applied to different sets of registers. These findings aresignificant given our need to make generalizations across the rapidlyincreasing number of corpora available for analysis.

 

Quick Read (beta)

loading the full paper ...