Understanding Learning Dynamics Of Language Models with SVCCA

  • 2018-11-01 04:51:20
  • Naomi Saphra, Adam Lopez
  • 20


Recent work has demonstrated that neural language models encode linguisticstructure implicitly in a number of ways. However, existing research has notshed light on the process by which this structure is acquired during training.We use SVCCA as a tool for understanding how a language model is implicitlypredicting a variety of word cluster tags. We present experiments suggestingthat a single recurrent layer of a language model learns linguistic structurein phases. We find, for example, that a language model naturally stabilizes itsrepresentation of part of speech earlier than it learns semantic and topicinformation.


Introduction (beta)



Conclusion (beta)