Abstract
The Mixture-of-Experts (MoE) layer, a sparsely-activated model controlled bya router, has achieved great success in deep learning. However, theunderstanding of such architecture remains elusive. In this paper, we formallystudy how the MoE layer improves the performance of neural network learning andwhy the mixture model will not collapse into a single model. Our empiricalresults suggest that the cluster structure of the underlying problem and thenon-linearity of the expert are pivotal to the success of MoE. To furtherunderstand this, we consider a challenging classification problem withintrinsic cluster structures, which is hard to learn using a single expert. Yetwith the MoE layer, by choosing the experts as two-layer nonlinearconvolutional neural networks (CNNs), we show that the problem can be learnedsuccessfully. Furthermore, our theory shows that the router can learn thecluster-center features, which helps divide the input complex problem intosimpler linear classification sub-problems that individual experts can conquer.To our knowledge, this is the first result towards formally understanding themechanism of the MoE layer for deep learning.