Large Margin Deep Networks for Classification

  • 2018-03-15 05:33:13
  • Gamaleldin F. Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, Samy Bengio
  • 39

Abstract

We present a formulation of deep learning that aims at producing a largemargin classifier. The notion of margin, minimum distance to a decisionboundary, has served as the foundation of several theoretically profound andempirically successful results for both classification and regression tasks.However, most large margin algorithms are applicable only to shallow modelswith a preset feature representation; and conventional margin methods forneural networks only enforce margin at the output layer. Such methods aretherefore not well suited for deep networks. In this work, we propose a novel loss function to impose a margin on anychosen set of layers of a deep network (including input and hidden layers). Ourformulation allows choosing any norm on the metric measuring the margin. Wedemonstrate that the decision boundary obtained by our loss has nice propertiescompared to standard classification loss functions. Specifically, we showimproved empirical results on the MNIST, CIFAR-10 and ImageNet datasets onmultiple tasks: generalization from small training sets, corrupted labels, androbustness against adversarial perturbations. The resulting loss is general andcomplementary to existing data augmentation (such as random/adversarial inputtransform) and regularization techniques (such as weight decay, dropout, andbatch norm).

 

Quick Read (beta)

loading the full paper ...