What's Hidden in a Randomly Weighted Neural Network?

  • 2019-11-29 18:56:53
  • Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari
  • 50

Abstract

Training a neural network is synonymous with learning the values of theweights. In contrast, we demonstrate that randomly weighted neural networkscontain subnetworks which achieve impressive performance without ever trainingthe weight values. Hidden in a randomly weighted Wide ResNet-50 we show thatthere is a subnetwork (with random weights) that is smaller than, but matchesthe performance of a ResNet-34 trained on ImageNet. Not only do these"untrained subnetworks" exist, but we provide an algorithm to effectively findthem. We empirically show that as randomly weighted neural networks with fixedweights grow wider and deeper, an "untrained subnetwork" approaches a networkwith learned weights in accuracy.

 

Quick Read (beta)

loading the full paper ...