Abstract
The success of neural language models (LMs) on many technological tasks hasbrought about their potential relevance as scientific theories of languagedespite some clear differences between LM training and child languageacquisition. In this paper we argue that some of the most prominent benchmarksfor evaluating the syntactic capacities of LMs may not be sufficientlyrigorous. In particular, we show that the template-based benchmarks lack thestructural diversity commonly found in the theoretical and psychologicalstudies of language. When trained on small-scale data modeling child languageacquisition, the LMs can be readily matched by simple baseline models. Weadvocate for the use of the readily available, carefully curated datasets thathave been evaluated for gradient acceptability by large pools of nativespeakers and are designed to probe the structural basis of grammarspecifically. On one such dataset, the LI-Adger dataset, LMs evaluate sentencesin a way inconsistent with human language users. We conclude with suggestionsfor better connecting LMs with the empirical study of child languageacquisition.