Critical Learning Periods in Deep Neural Networks

  • 2018-05-15 13:52:02
  • Alessandro Achille, Matteo Rovere, Stefano Soatto
  • 4

Abstract

Critical periods are phases in the early development of humans and animalsduring which experience can irreversibly affect the architecture of neuronalnetworks. In this work, we study the effects of visual stimulus deficits on thetraining of artificial neural networks (ANNs). Introducing well-characterizedvisual deficits, such as cataract-like blurring, in the early training phase ofa standard deep neural network causes a permanent performance loss that closelymimics critical period behavior in humans and animal models. Deficits that donot affect low-level image statistics, such as vertical flipping of the images,have no lasting effect on the ANNs' performance and can be rapidly overcomewith further training. In addition, the deeper the ANN is, the more pronouncedthe critical period. To better understand this phenomenon, we use FisherInformation as a measure of the strength of the network's connections duringthe training. Our information-theoretic analysis suggests that the first fewepochs are critical for the creation of strong connections across differentlayers, optimal for processing the input data distribution. Once such strongconnections are created, they do not appear to change during additionaltraining. These findings suggest that the initial rapid learning phase of ANNtraining, under-scrutinized compared to its asymptotic behavior, plays a keyrole in defining the final performance of networks. Our results also show howcritical periods are not restricted to biological systems, but can emergenaturally in learning systems, whether biological or artificial, due tofundamental constrains arising from learning dynamics and informationprocessing.

 

Introduction (beta)

None

 

Conclusion (beta)

None