Abstract
Natural Language Processing (NLP) has become one of the leading applicationareas in the current Artificial Intelligence boom. Transfer learning hasenabled large deep learning neural networks trained on the language modelingtask to vastly improve performance in almost all downstream language tasks.Interestingly, when the language models are trained with data that includessoftware code, they demonstrate remarkable abilities in generating functioningcomputer code from natural language specifications. We argue that this createsa conundrum for the claim that eliminative neural models are a radicalrestructuring in our understanding of cognition in that they eliminate the needfor symbolic abstractions like generative phrase structure grammars. Becausethe syntax of programming languages is by design determined by phrase structuregrammars, neural models that produce syntactic code are apparentlyuninformative about the theoretical foundations of programming languages. Thedemonstration that neural models perform well on tasks that involve clearlysymbolic systems, proves that they cannot be used as an argument that languageand other cognitive systems are not symbolic. Finally, we argue as a corollarythat the term language model is misleading and propose the adoption of theworking term corpus model instead, which better reflects the genesis andcontents of the model.