Abstract
Do language models (LMs) offer insights into human language learning? Acommon argument against this idea is that because their architecture andtraining paradigm are so vastly different from humans, LMs can learn arbitraryinputs as easily as natural languages. We test this claim by training LMs tomodel impossible and typologically unattested languages. Unlike previous work,which has focused exclusively on English, we conduct experiments on 12languages from 4 language families with two newly constructed parallel corpora.Our results show that while GPT-2 small can largely distinguish attestedlanguages from their impossible counterparts, it does not achieve perfectseparation between all the attested languages and all the impossible ones. Wefurther test whether GPT-2 small distinguishes typologically attested fromunattested languages with different NP orders by manipulating word order basedon Greenberg's Universal 20. We find that the model's perplexity scores do notdistinguish attested vs. unattested word orders, while its performance on thegeneralization test does. These findings suggest that LMs exhibit somehuman-like inductive biases, though these biases are weaker than those found inhuman learners.