Using Inherent Structures to design Lean 2-layer RBMs

  • 2018-06-12 14:55:42
  • Abhishek Bansal, Abhinav Anand, Chiranjib Bhattacharyya
  • 1


Understanding the representational power of Restricted Boltzmann Machines(RBMs) with multiple layers is an ill-understood problem and is an area ofactive research. Motivated from the approach of \emph{Inherent Structureformalism} (Stillinger & Weber, 1982), extensively used in analysing SpinGlasses, we propose a novel measure called \emph{Inherent Structure Capacity}(ISC), which characterizes the representation capacity of a fixed architectureRBM by the expected number of modes of distributions emanating from the RBMwith parameters drawn from a prior distribution. Though ISC is intractable, weshow that for a single layer RBM architecture ISC approaches a finite constantas number of hidden units are increased and to further improve the ISC, oneneeds to add a second layer. Furthermore, we introduce \emph{Lean} RBMs, whichare multi-layer RBMs where each layer can have at-most $O(n)$ units with thenumber of visible units being n. We show that for every single layer RBM with$\Omega(n^{2+r}), r \ge 0$, hidden units there exists a two-layered \emph{lean}RBM with $\Theta(n^2)$ parameters with the same ISC, establishing that 2 layerRBMs can achieve the same representational power as single-layer RBMs but usingfar fewer number of parameters. To the best of our knowledge, this is the firstresult which quantitatively establishes the need for layering.


Introduction (beta)



Conclusion (beta)