Continuous Learning in a Hierarchical Multiscale Neural Network

  • 2018-05-15 13:37:33
  • Thomas Wolf, Julien Chaumond, Clement Delangue
  • 7

Abstract

We reformulate the problem of encoding a multi-scale representation of asequence in a language model by casting it in a continuous learning framework.We propose a hierarchical multi-scale language model in which short time-scaledependencies are encoded in the hidden state of a lower-level recurrent neuralnetwork while longer time-scale dependencies are encoded in the dynamic of thelower-level network by having a meta-learner update the weights of thelower-level neural network in an online meta-learning fashion. We use elasticweights consolidation as a higher-level to prevent catastrophic forgetting inour continuous learning framework.

 

Introduction (beta)

None

 

Conclusion (beta)

None