Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures

  • 2018-07-12 12:53:50
  • Sergey Bartunov, Adam Santoro, Blake A. Richards, Geoffrey E. Hinton, Timothy Lillicrap
  • 199

Abstract

The backpropagation of error algorithm (BP) is often said to be impossible toimplement in a real brain. The recent success of deep networks in machinelearning and AI, however, has inspired proposals for understanding how thebrain might learn across multiple layers, and hence how it might implement orapproximate BP. As of yet, none of these proposals have been rigorouslyevaluated on tasks where BP-guided deep learning has proved critical, or inarchitectures more structured than simple fully-connected networks. Here wepresent the first results on scaling up biologically motivated models of deeplearning on datasets which need deep networks with appropriate architectures toachieve good performance. We present results on the MNIST, CIFAR-10, andImageNet datasets and explore variants of target-propagation (TP) and feedbackalignment (FA) algorithms, and explore performance in both fully- andlocally-connected architectures. We also introduce weight-transport-freevariants of difference target propagation (DTP) modified to removebackpropagation from the penultimate layer. Many of these algorithms performwell for MNIST, but for CIFAR and ImageNet we find that TP and FA variantsperform significantly worse than BP, especially for networks composed oflocally connected units, opening questions about whether new architectures andalgorithms are required to scale these approaches. Our results andimplementation details help establish baselines for biologically motivated deeplearning schemes going forward.

 

Quick Read (beta)

loading the full paper ...