Ancestor-to-Creole Transfer is Not a Walk in the Park

  • 2022-06-09 10:28:10
  • Heather Lent, Emanuele Bugliarello, Anders S√łgaard
  • 6

Abstract

We aim to learn language models for Creole languages for which large volumesof data are not readily available, and therefore explore the potential transferfrom ancestor languages (the 'Ancestry Transfer Hypothesis'). We find thatstandard transfer methods do not facilitate ancestry transfer. Surprisingly,different from other non-Creole languages, a very distinct two-phase patternemerges for Creoles: As our training losses plateau, and language models beginto overfit on their source languages, perplexity on the Creoles drop. Weexplore if this compression phase can lead to practically useful languagemodels (the 'Ancestry Bottleneck Hypothesis'), but also falsify this. Moreover,we show that Creoles even exhibit this two-phase pattern even when training onrandom, unrelated languages. Thus Creoles seem to be typological outliers andwe speculate whether there is a link between the two observations.

 

Quick Read (beta)

loading the full paper ...