Abstract

Recent work has demonstrated that neural language models encode linguisticstructure implicitly in a number of ways. However, existing research has notshed light on the process by which this structure is acquired during training.We use SVCCA as a tool for understanding how a language model is implicitlypredicting a variety of word cluster tags. We present experiments suggestingthat a single recurrent layer of a language model learns linguistic structurein phases. We find, for example, that a language model naturally stabilizes itsrepresentation of part of speech earlier than it learns semantic and topicinformation.

 

Quick Read (beta)