Using Inherent Structures to design Lean 2-layer RBMs

  • 2018-06-12 14:55:42
  • Abhishek Bansal, Abhinav Anand, Chiranjib Bhattacharyya
  • 1

Abstract

Understanding the representational power of Restricted Boltzmann Machines(RBMs) with multiple layers is an ill-understood problem and is an area ofactive research. Motivated from the approach of \emph{Inherent Structureformalism} (Stillinger & Weber, 1982), extensively used in analysing SpinGlasses, we propose a novel measure called \emph{Inherent Structure Capacity}(ISC), which characterizes the representation capacity of a fixed architectureRBM by the expected number of modes of distributions emanating from the RBMwith parameters drawn from a prior distribution. Though ISC is intractable, weshow that for a single layer RBM architecture ISC approaches a finite constantas number of hidden units are increased and to further improve the ISC, oneneeds to add a second layer. Furthermore, we introduce \emph{Lean} RBMs, whichare multi-layer RBMs where each layer can have at-most $O(n)$ units with thenumber of visible units being n. We show that for every single layer RBM with$\Omega(n^{2+r}), r \ge 0$, hidden units there exists a two-layered \emph{lean}RBM with $\Theta(n^2)$ parameters with the same ISC, establishing that 2 layerRBMs can achieve the same representational power as single-layer RBMs but usingfar fewer number of parameters. To the best of our knowledge, this is the firstresult which quantitatively establishes the need for layering.

 

Introduction (beta)

None

 

Conclusion (beta)

None