Penalty Method for Inversion-Free Deep Bilevel Optimization

  • 2020-02-20 18:42:21
  • Akshay Mehra, Jihun Hamm
  • 0

Abstract

Bilevel optimization problems are at the center of several important machinelearning problems such as hyperparameter tuning, data denoising, meta- andfew-shot learning, data poisoning. Different from simultaneous ormulti-objective optimization, bilevel optimization requires computing theinverse of the Hessian of the lower-level cost function to obtain the exactdescent direction for the upper-level cost. In this paper, we propose a newmethod for solving deep bilevel optimization problems using the penaltyfunction which avoids computing the inverse. We prove convergence of our methodunder mild conditions and show that it computes the exact hypergradientasymptotically. Small space and time complexity of our method enables us tosolve large-scale bilevel problems involving deep neural networks with severalmillion parameters. We present results of our method for data denoising onMNIST/CIFAR10/SVHN datasets, for few-shot learning on Omniglot/Mini-Imagenetdatasets and for training-data poisoning on MNIST/Imagenet datasets. In allexperiments, our method outperforms or is comparable to previously proposedmethods both in terms of accuracy and run-time.

 

Quick Read (beta)

loading the full paper ...