Using linear initialisation to improve speed of convergence and fully-trained error in Autoencoders

  • 2023-11-17 18:43:32
  • Marcel Marais, Mate Hartstein, George Cevora
  • 0


Good weight initialisation is an important step in successful training ofArtificial Neural Networks. Over time a number of improvements have beenproposed to this process. In this paper we introduce a novel weightinitialisation technique called the Straddled Matrix Initialiser. Thisinitialisation technique is motivated by our assumption that major,global-scale relationships in data are linear with only smaller effectsrequiring complex non-linearities. Combination of Straddled Matrix and ReLUactivation function initialises a Neural Network as a de facto linear model,which we postulate should be a better starting point for optimisation given ourassumptions. We test this by training autoencoders on three datasets usingStraddled Matrix and seven other state-of-the-art weight initialisationtechniques. In all our experiments the Straddeled Matrix Initialiser clearlyoutperforms all other methods.


Quick Read (beta)

loading the full paper ...