Abstract
Good weight initialisation is an important step in successful training ofArtificial Neural Networks. Over time a number of improvements have beenproposed to this process. In this paper we introduce a novel weightinitialisation technique called the Straddled Matrix Initialiser. Thisinitialisation technique is motivated by our assumption that major,global-scale relationships in data are linear with only smaller effectsrequiring complex non-linearities. Combination of Straddled Matrix and ReLUactivation function initialises a Neural Network as a de facto linear model,which we postulate should be a better starting point for optimisation given ourassumptions. We test this by training autoencoders on three datasets usingStraddled Matrix and seven other state-of-the-art weight initialisationtechniques. In all our experiments the Straddeled Matrix Initialiser clearlyoutperforms all other methods.