The emergent algebraic structure of RNNs and embeddings in NLP

  • 2018-03-07 19:06:08
  • Sean A. Cantrell
  • 40


We examine the algebraic and geometric properties of a uni-directional GRUand word embeddings trained end-to-end on a text classification task. Ahyperparameter search over word embedding dimension, GRU hidden dimension, anda linear combination of the GRU outputs is performed. We conclude that wordsnaturally embed themselves in a Lie group and that RNNs form a nonlinearrepresentation of the group. Appealing to these results, we propose a novelclass of recurrent-like neural networks and a word embedding scheme.


Introduction (beta)



Conclusion (beta)