Neural Network Encapsulation

  • 2018-08-11 04:36:53
  • Hongyang Li, Xiaoyang Guo, Bo Dai, Wanli Ouyang, Xiaogang Wang
  • 59

Abstract

A capsule is a collection of neurons which represents different variants of apattern in the network. The routing scheme ensures only certain capsules whichresemble lower counterparts in the higher layer should be activated. However,the computational complexity becomes a bottleneck for scaling up to largernetworks, as lower capsules need to correspond to each and every highercapsule. To resolve this limitation, we approximate the routing process withtwo branches: a master branch which collects primary information from itsdirect contact in the lower layer and an aide branch that replenishes masterbased on pattern variants encoded in other lower capsules. Compared withprevious iterative and unsupervised routing scheme, these two branches arecommunicated in a fast, supervised and one-time pass fashion. The complexityand runtime of the model are therefore decreased by a large margin. Motivatedby the routing to make higher capsule have agreement with lower capsule, weextend the mechanism as a compensation for the rapid loss of information innearby layers. We devise a feedback agreement unit to send back higher capsulesas feedback. It could be regarded as an additional regularization to thenetwork. The feedback agreement is achieved by comparing the optimal transportdivergence between two distributions (lower and higher capsules). Such anadd-on witnesses a unanimous gain in both capsule and vanilla networks. Ourproposed EncapNet performs favorably better against previous state-of-the-artson CIFAR10/100, SVHN and a subset of ImageNet.

 

Quick Read (beta)

loading the full paper ...