A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks

  • 2019-07-05 01:49:45
  • Owen Marschall, Kyunghyun Cho, Cristina Savin
  • 24


We present a framework for compactly summarizing many recent results inefficient and/or biologically plausible online training of recurrent neuralnetworks (RNN). The framework organizes algorithms according to severalcriteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs.deterministic, and (d) closed form vs. numerical. These axes reveal latentconceptual connections among several recent advances in online learning.Furthermore, we provide novel mathematical intuitions for their degree ofsuccess. Testing various algorithms on two synthetic tasks shows thatperformances cluster according to our criteria. Although a similar clusteringis also observed for gradient alignment, alignment with exact methods does notalone explain ultimate performance, especially for stochastic algorithms. Thissuggests the need for better comparison metrics.


Quick Read (beta)

A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks

\nameOwen Marschall \email[email protected]
\addrCenter for Neural Science
New York University
New York, NY 10006, USA \AND\nameKyunghyun Cho \email[email protected]
\addrNew York University
Facebook AI Research
CIFAR Azrieli Global Scholar \AND\nameCristina Savin \email[email protected]
\addrCenter for Neural Science
Center for Data Science
New York University

We present a framework for compactly summarizing many recent results in efficient and/or biologically plausible online training of recurrent neural networks (RNN). The framework organizes algorithms according to several criteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs. deterministic, and (d) closed form vs. numerical. These axes reveal latent conceptual connections among several recent advances in online learning. Furthermore, we provide novel mathematical intuitions for their degree of success. Testing various algorithms on two synthetic tasks shows that performances cluster according to our criteria. Although a similar clustering is also observed for gradient alignment, alignment with exact methods does not alone explain ultimate performance, especially for stochastic algorithms. This suggests the need for better comparison metrics.

A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks Owen Marschall [email protected]
Center for Neural Science
New York University
New York, NY 10006, USA
Kyunghyun Cho [email protected]
New York University
Facebook AI Research
CIFAR Azrieli Global Scholar
Cristina Savin [email protected]
Center for Neural Science
Center for Data Science
New York University


Keywords: real-time recurrent learning, backpropagation through time, approximation, biologically plausible learning, local, online

1 Introduction

Training recurrent neural networks (RNN) to learn sequence data is traditionally done with stochastic gradient descent (SGD), using the backpropagation through time algorithm (BPTT, Werbos et al., 1990) to calculate the gradient. This requires “unrolling” the network over some range of time steps T and performing backpropagation as though the network were feedforward under the constraint of sharing parameters across time steps (“layers”). BPTT’s success in a wide range of applications (Mikolov et al., 2010; Graves, 2013; Bahdanau et al., 2016, 2014; Cho et al., 2015; Graves et al., 2016) has made it the industry standard; however, there exist alternative online algorithms for training RNNs. These compute gradients in real time as the network runs forward, without explicitly referencing past activity or averaging over batches of data. There are two reasons for considering online alternatives to BPTT. One is practical: computational costs do not scale with T. The other is conceptual: human brains are able to learn long-term dependencies without explicitly memorizing all past brain states, and understanding online learning is a key step in the larger project of understanding human learning.

The classic online learning algorithm is real-time recurrent learning (RTRL, Williams and Zipser, 1989), which is equivalent to BPTT in the limit of a small learning rate (Murray, 2019). RTRL recursively updates the total derivative of the hidden state with respect to the parameters, eliminating the need to reference past activity but introducing an order n (hidden units) ×n2 (parameters) =n3 memory requirement. In practice, this is often more computationally demanding than BPTT (order nT), hence not frequently used in applications. Nor is RTRL at face value a good model of biological learning, for the same reason: no known biological mechanism exists to store—let alone manipulate—a float for each synapse-neuron pair. Thus RTRL and online learning more broadly have remained relatively obscure footnotes to both the deep learning revolution itself and its impact on computational neuroscience.

Recent advances in recurrent network architectures have brought the issue of online learning back into the spotlight. While vanishing gradients used to significantly limit the extent of the temporal dependencies that an RNN could learn, new architectures like LSTMs (Hochreiter and Schmidhuber, 1997) and GRUs (Cho et al., 2014) have dramatically expanded this learnable time horizon. Unfortunately, taking advantage of this capacity requires an equally dramatic expansion in computational resources, if using BPTT. This has led to an explosion of novel online learning algorithms (Tallec and Ollivier, 2017; Mujika et al., 2018; Roth et al., 2019; Murray, 2019; Jaderberg et al., 2017) which aim to improve on the efficiency of RTRL, in many cases using update rules that might be implemented by a biological circuit.

The sheer number and variety of these approaches pose challenges for both theory and practice. It is not always completely clear what makes various algorithms different from one another, how they are conceptually related, or even why they might work in the first place. There is a pressing need in the field for a cohesive framework for describing and comparing online methods. Here we aim to provide a thorough overview of modern online algorithms for training RNNs, in a way that provides a clearer understanding of the mathematical structure underlying all these different approaches. Our framework organizes the existing literature along several axes that encode meaningful conceptual distinctions:

  1. a)

    Past facing vs. future facing

  2. b)

    The tensor structure of the algorithm

  3. c)

    Stochastic vs. deterministic update

  4. d)

    Closed form vs. numerical solution for update

These axes will be explained in detail later, but briefly: the past vs. future axis is a root distinction that divides algorithms by the type of gradient they calculate, while the other three describe their representations and update principles. Table 1 contains (to our knowledge) all recently published online learning algorithms for RNNs, categorized according to these criteria. We can already see that many combinations of these characteristics manifest in the literature, suggesting that new algorithms could be developed by mixing and matching properties. (We provide a concrete example of this in §3.4.)

Algorithm Facing Tensor Update Memory Time
RTRL Past Mkij Deterministic Closed-form n3 n4
UORO Past AkBij Stochastic Closed-form n2 n2
KF-RTRL Past AjBki Stochastic Closed-form n2 n3
R-KF-RTRL Past AiBjk Stochastic Closed-form n2 n3
r-OK Past l=1rAljBlki Stochastic Numerical rn2 rn3
KeRNL Past AkiBij Deterministic Numerical n2 n2
RFLO Past δkiBij Deterministic Closed-form n2 n2
E-BPTT Deterministic Closed-form nT n2
F-BPTT Future Deterministic Closed-form nT n2T
DNI Future Ali Deterministic Numerical n2 n2
Table 1: A list of learning algorithms reviewed here, together with their main properties. The indices k, i and j reference different dimensions of the “influence tensor” of RTRL (§3.1.1); l references components of the feedback vector 𝐚~ in DNI (§4.2).

Here we describe each algorithm in unified notation that makes clear their classification by these criteria. In the process, we generate novel intuitions about why different approximations can be successful and discuss some of the finer points of their biological plausibility. Finally, we simulate each algorithm on a common set of synthetic tasks with vanilla RNN architecture for simplicity. We compare performance and analyze gradient alignments to see to what extent their empirical similarity is predicted by their similarity according to our framework. Algorithm performance roughly clusters according to criteria (a)-(d) across tasks, lending credence to our approach. Curiously, gradient alignment with exact methods (RTRL and BPTT) does not predict performance, despite its ubiquity as a tool for analyzing approximate learning algorithms.

2 Past- and future-facing perspectives of online learning

Before we dive into the details of these algorithms, we first articulate what we mean by past- and future-facing, related to the “reverse/forward accumulation” distinction concurrently described by Cooijmans and Martens (2019). Consider a recurrent neural network that contains, at each time step t, a state 𝐚(t)n. This state is updated via a function F𝐰:mn, which is parameterized by a flattened vector of parameters 𝐰P. Here m=n+nin+1 counts the total number of input dimensions, including the recurrent inputs 𝐚(t-1)n, task inputs 𝐱(t)nin, and an additional input clamped to 1 (to represent bias). For some initial state 𝐚(0), F𝐰 defines the network dynamics by


At each time step an output 𝐲(t)nout is computed by another function F𝐰oout:nnout, parameterized by 𝐰oPo. We will typically choose an affine-softmax readout for F𝐰oout, with output weights/bias 𝐖outnout×(n+1). A loss function L(𝐲(t),𝐲*(t)) calculates an instantaneous loss L(t), quantifying to what degree the predicted output 𝐲(t) matches the target output 𝐲*(t).

The goal is to train the network by gradient descent (or other gradient-based optimizers such as ADAM from Kingma and Ba, 2014) on the total loss =L(t) w.r.t. the parameters 𝐰 and 𝐰o. It is natural to learn 𝐰o online, because only information at present time t is required to calculate the gradient L(t)/𝐰o. So the heart of the problem is to calculate /𝐰.

The parameter 𝐰 is applied via F𝐰 at every time step, and we denote a particular application of 𝐰 at time s as 𝐰(s). Of course, a recurrent system is constrained to share parameters across time steps, so a perturbation δ𝐰 is effectively a perturbation across all applications δ𝐰(s), i.e., 𝐰(s)/𝐰=𝐈P. In principle, each application of the parameters affects all future losses L(t), ts. The core of any recurrent learning algorithm is to estimate the influence L(t)/𝐰(s) of one parameter application 𝐰(s) on one loss L(t), since these individual terms are necessary and sufficient to define the global gradient

𝐰=tL(t)𝐰=tstL(t)𝐰(s)𝐰(s)𝐰=tstL(t)𝐰(s). (1)

This raises the question of how to sum these components to produce individual gradients to pass to the optimizer. In truncated BPTT, one unrolls the graph over some range of time steps and sums L(t)/𝐰(s) for all t,s in that range with ts (see §4.1.1). This does not qualify as an “online” learning rule, because it requires two independent time indices—at most one can represent “real time” leaving the other to represent the future or the past. If we can account for one of the summations via dynamic updates, then the algorithm is online or temporally local, i.e. not requiring explicit reference to the past or future. As depicted in Fig. 1, there are two possibilities. If t from Eq. (1) corresponds to real time, then the gradient passed to the optimizer is

𝐰(t)=s=0tL(t)𝐰(s)=L(t)𝐰. (2)

In this case, we say learning is past facing, because the gradient is a sum of the influences of past applications of 𝐰 on the current loss. On the other hand, if s from Eq. (1) represents real time, then the gradient passed to the optimizer is

𝐰(s)=t=sL(t)𝐰(s)=𝐰(s). (3)

Here we say learning is future facing, because the gradient is a sum of influences by the current application of 𝐰 on future losses.

Figure 1: Cartoon depicting the past- and future-facing perspectives of online learning, for an RNN unrolled over time. Each 𝐚 represents the RNN hidden state value, while F𝐰 denotes applications of the recurrent update; the instantaneous losses L implicitly depend on the hidden state through L(t)=L(F𝐰oout(𝐚(t)),𝐲*(t)). The blue (yellow) arrows show the paths of influence accounted for by the past-facing (future-facing) gradient described in the corresponding equation.

2.1 Past-facing online learning algorithms

Here we derive a fundamental relation leveraged by past-facing (PF) online algorithms. Let t index real time, and define the influence matrix 𝐌(t)n×P, where n and P are respectively the number of hidden units and the number of parameters defining F𝐰. 𝐌(t) tracks the derivatives of the current state 𝐚(t) with respect to each parameter wp:

Mkp(t)=ak(t)wp. (4)

Let’s rewrite Eq. (4) with matrix notation and unpack it by one time step:

𝐌(t) =𝐚(t)𝐰=st𝐚(t)𝐰(s)=st-1𝐚(t)𝐰(s)+𝐚(t)𝐰(t)
𝐉(t)𝐌(t-1)+𝐌¯(t). (5)

A simple recursive formula emerges, wherein the influence matrix is updated by multiplying its current value by the Jacobian 𝐉(t)=𝐚(t)/𝐚(t-1)n×n of the network and then adding the immediate influence 𝐌¯(t)=𝐚(t)/𝐰(t)n×P. To compute the gradient that ultimately gets passed to the optimizer, we simply use the chain rule over the current hidden state 𝐚(t):

L(t)𝐰=L(t)𝐚(t)𝐚(t)𝐰𝐜¯(t)𝐌(t), (6)

where the immediate credit assignment vector 𝐜¯(t)n is defined to be L(t)/𝐚(t) and is calculated by backpropagating the error 𝜹(t) through the derivative of the output function F𝐰oout (or approximated by Feedback Alignment, see Lillicrap et al., 2016). In the end, we compute a derivative in Eq. (6) that is implicitly a sum over the many terms of Eq. (2), using formulae that depend explicitly only on times t and t-1. For this reason, such a learning algorithm is online, and it is past facing because the gradient computation is of the form in Eq. (2).

2.2 Future-facing online learning algorithms

Here we show a symmetric relation for future-facing (FF) online algorithms. The credit assignment vector 𝐜(t)n is a row vector defined as the gradient of the loss with respect to the hidden state 𝐚(t). It plays a role analogous to 𝐌(t) and has a recursive update similar to Eq. (5):

𝐜(t) =𝐚(t)=stL(s)𝐚(t)=L(t)𝐚(t)+st+1L(s)𝐚(t)
=𝐜¯(t)+𝐜(t+1)𝐉(t+1). (7)

As in the PF case, the gradient is ultimately calculated using the chain rule over 𝐚(t):

𝐰(t)=𝐚(t)𝐚(t)𝐰(t)𝐜(t)𝐌¯(t). (8)

The recursive relations for PF and FF algorithms are of identical form given the following changes: (1) swap the roles of and 𝐰, (2) swap the roles of t-1 and t+1, and (3) flip the direction of all derivatives. This clarifies the fundamental trade-off between the PF and FF approaches to online learning. On the one hand, memory requirements favor FF because is a scalar while 𝐰 is a matrix. On the other, only PF can truly be run online, because the time direction of the update in FF is opposite the forward pass. Thus, efficient PF algorithms must compress 𝐌(t), while efficient FF algorithms must predict 𝐜(t+1).

3 Past-facing algorithms

3.1 Real-Time Recurrent Learning

The Real-Time Recurrent Learning (RTRL, Williams and Zipser, 1989) algorithm directly applies Eqs. (5) and (6) as written. We call the application of Eq. (5) the “update” to the learning algorithm, which is deterministic and in closed form. Implementing Eq. (5) requires storing nP𝒪(n3) floats in 𝐌(t) and performing 𝒪(n4) multiplications in 𝐉(t)𝐌(t), which is neither especially efficient nor biologically plausible. However, several efficient (and in some cases, biologically plausible) online learning algorithms have recently been developed, including Unbiased Online Recurrent Optimization (UORO; Tallec and Ollivier, 2017; §3.2), Kronecker-Factored RTRL (KF-RTRL; Mujika et al., 2018; §3.3), Kernel RNN Learning (KeRNL; Roth et al., 2019; §3.5), and Random-Feedback Online Learning (RFLO; Murray, 2019; §3.6). We claim that these learning algorithms, whether explicitly derived as such or not, are all implicitly approximations to RTRL, each a special case of a general class of techniques for compressing 𝐌(t). In the following section, we clarify how each of these learning algorithms fits into this broad structure.

3.1.1 Approximations to RTRL

To concretely illuminate these ideas, we will work with a special case of F𝐰, a time-continuous vanilla RNN:

𝐚(t)=F𝐰(𝐚(t-1),𝐱(t))=(1-α)𝐚(t-1)+αϕ(𝐖𝐚^(t-1)), (9)

where 𝐚^(t-1)=concat(𝐚(t-1),𝐱(t),1)m, 𝐖n×m, ϕ:nn is some point-wise nonlinearity (e.g. tanh), and α(0,1] is the network’s inverse time constant. The trainable parameters wp are folded via the indexing p=i×n+j into the weight matrix Wij, whose columns hold the recurrent weights, the input weights, and a bias. By reshaping wp into its natural matrix form Wij, we can write the influence matrix as an order-3 influence tensor


Thus Mkij(t) specifies the effect on the k-th unit of perturbing the direct connection from the j-th unit to the i-th unit. The immediate influence can also be written as a tensor. By differentiating Eq. (9), we see it takes the sparse form


because Wij can affect the k-th unit directly only if k=i. Many approximations of RTRL involve a decomposition of Mkij(t) into a product of lower-order tensors. For example, UORO represents Mkij(t) by an outer product Ak(t)Bij(t), which has a memory requirement of only 𝒪(n2). Similarly, KF-RTRL uses a Kronecker-product decomposition Aj(t)Bki(t). We can generalize these cases into a set of six possible decompositions of Mkij(t) into products of lower-order tensors A(t) and B(t):

Mkij(t){Ak(t)Bij(t)UORO,§3.2Aj(t)Bki(t)KF-RTRL,§3.3Ai(t)Bkj(t)“Reverse” KF-RTRL,§3.4Aki(t)Bij(t)KeRNL/RFLO,§3.5/§3.6Akj(t)Bij(t)UnexploredAki(t)Bkj(t)Unexplored.

Each such decomposition has a memory requirement of 𝒪(n2). Of course, it is not sufficient to write down an idealized decomposition for a particular time point; there must exist some efficient way to update the decomposition as the network runs forwards. We now go through each algorithm and show the mathematical techniques used to derive update equations and categorize them by the criteria outlined in Table 1.

3.2 Unbiased Online Recurrent Optimization (UORO)

Tallec and Ollivier (2017) discovered a technique for approximating 𝐌(t)n×P as an outer product 𝐀(t)𝐁(t), where 𝐀(t)n×1 and 𝐁(t)1×P. The authors proved a crucial lemma (see Appendix A or Tallec and Ollivier, 2017) that gives, in closed form, an unbiased rank-1 estimate of a given matrix over the choice of a random vector 𝝂n with 𝔼[νiνj]δij and 𝔼[νi]=0. They leverage this result to derive a closed-form update rule for 𝐀(t) and 𝐁(t) at each time step, without ever having to explicitly (and expensively) calculate 𝐌(t). We present an equivalent formulation in terms of tensor components, i.e.,


where Bij(t) represents the “rolled-up” components of 𝐁(t), as in Wij w.r.t. 𝐰. Intuitively, the kij-th component of the influence matrix is constrained to be the product of the k-th unit’s “sensitivity” Ak(t) and the ij-th parameter’s “efficacy” Bij(t). Eqs. (10) and (11) show the form of the update and why it is unbiased over 𝝂, respectively:

Ak(t)Bij(t) =(ρ0kJkk(t)Ak(t-1)+ρ1νk)(ρ0-1Bij(t-1)+ρ1-1kνkM¯kij(t))
+kνk[ρ1ρ0-1δkkBij(t-1)+ρ0ρ1-1M¯kij(t)k′′Jkk′′(t)Ak′′(t-1)] (10)
𝔼[Ak(t)Bij(t)] =kJkk(t)𝔼[Ak(t-1)Bij(t-1)]+k𝔼[νkνk]M¯kij(t)
+k𝔼[νk](cross terms)
=kJkk(t)Mkij(t-1)+kδkkM¯kij(t)+k0×(cross terms)
=Mkij(t). (11)

The cross terms vanish in expectation because 𝔼[νk]=0. Thus, by induction over t, the estimate of Mkij(t) remains unbiased at every time step. The constants ρ0,ρ1>0 are chosen at each time step to minimize total variance of the estimate by balancing the norms of the cross terms. This algorithm’s update is stochastic due to its reliance on the random vector 𝝂, but it is in closed form because it has an explicit update formula (Eq. 10). Both its memory and computational complexity are 𝒪(n2).

3.3 Kronecker-Factored RTRL (KF-RTRL)

Mujika et al. (2018) leverage the same lemma as in UORO, but using a decomposition of 𝐌(t) in terms of a Kronecker product 𝐀(t)𝐁(t), where now 𝐀(t)1×m and 𝐁(t)n×n. This decomposition is more natural, because the immediate influence 𝐌¯(t) factors exactly as a Kronecker product 𝐚^(t)𝐃(t) for vanilla RNNs, where Dki(t)=αδkiϕ(hi(t)). To derive the update rule for UORO, one must first generate a rank-1 estimate of 𝐌¯(t) as an intermediate step, introducing more variance, but in KF-RTRL, this step is unnecessary. In terms of components, the compression takes the form


which is similar to UORO, modulo a cyclic permutation of the indices. Given a sample 𝝂2 of only 2 i.i.d. random variables, again with 𝔼[νiνj]δij and 𝔼[νi]=0, the update takes the form shown in Eqs. (12) and (13):

Aj(t) =(ν0ρ0Aj(t-1)+ν1ρ1a^j(t-1)) (12)
Bki(t) =(ν0ρ0-1kJkk(t)Bki(t-1)+ν1ρ1-1αδkiϕ(hi(t))) (13)
Aj(t)Bki(t) =ν02kJkk(t)Aj(t-1)Bki(t-1)+ν12αδkiϕ(hi(t))a^j(t-1)+cross-terms
𝔼[Aj(t)Bki(t)] =kJkk(t)𝔼[Aj(t-1)Bki(t-1)]+αδkiϕ(hi(t))a^j(t-1)

As in UORO, the cross terms vanish in expectation, and the estimate is unbiased by induction over t. This algorithm’s updates are also stochastic and in closed form. Its memory complexity is 𝒪(n2), but its computation time is 𝒪(n3) because of the matrix-matrix product in Eq. (13).

3.4 Reverse KF-RTRL (R-KF-RTRL)

Our exploration of the space of different approximations naturally raises a question: is an approximation of the form

Mkij(t)Ai(t)Bkj(t) (14)

also possible? We refer to this method as “Reverse” KF-RTRL (R-KF-RTRL) because, in matrix notation, this would be formulated as 𝐌(t)𝐁(t)𝐀(t), where 𝐀(t)1×n and 𝐁(t)n×m. We propose the following update for Ai(t) and Bkj(t) in terms of a random vector 𝝂n:

Ai(t)Bkj(t) =(ρ0Ai(t-1)+ρ1νi)(ρ0-1kJkk(t)Bkj(t-1)+ρ1-1iνiM¯kij(t)) (15)
𝔼[Ai(t)Bkj(t)] =kJkk(t)𝔼[Ai(t-1)Bkj(t-1)]+M¯kij(t)
=Mkij(t). (16)

Eq. (16) shows that this estimate is unbiased, using updates that are stochastic and in closed form, like its sibling algorithms. Its memory and computational complexity are 𝒪(n2) and 𝒪(n3), respectively. R-KF-RTRL is actually more similar to UORO than KF-RTRL, because M¯kij(t) does not naturally factor like Eq. (14), introducing more variance. Worse, it has the computational complexity of KF-RTRL due to the matrix-matrix multiplication in Eq. (15). KF-RTRL stands out as the most effective of these 3 algorithms, because it estimates 𝐌(t) with the lowest variance due to its natural decomposition structure. (See Mujika et al., 2018 for variance calculations.)

3.4.1 Optimal Kronecker-Sum Approximation (OK)

We briefly mention an extension of KF-RTRL by Benzing et al. (2019), where the influence matrix is approximated not by 1 but rather a sum of r Kronecker products, or, in components


On the RTRL update, the k index of Blki(t) is propagated forward by the Jacobian, and then the immediate influence—itself a Kronecker product—is added. Now Mkij(t) is approximated by r+1 Kronecker products


but the authors developed a technique to optimally reduce this sum back to r Kronecker products, keeping the memory complexity 𝒪(rn2) and computational complexity 𝒪(rn3) constant. This update is stochastic because it requires explicit randomness in the flavor of the above algorithms, and it is numerical because there is no closed form solution to the update. We leave the details to the original paper.

3.5 Kernel RNN Learning (KeRNL)

Roth et al. (2019) developed a learning algorithm for RNNs that is essentially a compression of the influence matrix of the form Mkij(t)AkiBij(t). We will show that this algorithm is also an implicit approximation of RTRL, although the update rules are fundamentally different than those for UORO, KF-RTRL and R-KF-RTRL. The eligibility trace 𝐁(t)n×m updates by temporally filtering the immediate influences αϕ(hi(t))a^j(t-1) with unit-specific, learnable timescales αi:

Bij(t)=(1-αi)Bij(t-1)+αϕ(hi(t))a^j(t-1). (17)

The sensitivity matrix 𝐀n×n is chosen to approximate the multi-step Jacobian ak(t)/ai(t) with help from the learned timescales:

ak(t)ai(t)Aki(1-αi)(t-t). (18)

We will describe how 𝐀 is learned later, but for now we assume this approximation holds and use it to show how the KeRNL update is equivalent to that of RTRL. We have dropped the explicit time-dependence from 𝐀, because it updates too slowly for Eq. (18) to be specific to any one time point. If we unpack this approximation by one time step, we uncover the consistency relation

Aki(1-αi)kJkk(t)Aki. (19)

By taking t=t in Eq. (18) and rearranging Eq. (19), we see this approximation implicitly assumes both

Aki{δki(1-αi)-1kJkk(t)Aki. (20)

Then the eligibility trace update effectively implements the RTRL update, assuming inductively that Mkij(t-1) is well approximated by AkiBij(t-1):

AkiBij(t) =Aki[(1-αi)Bij(t-1)+αϕ(hi(t))a^j(t-1)]
kJkk(t)AkiBij(t-1)+αδkiϕ(hi(t))a^j(t-1) (21)

In Eq. (21), we use each of the special cases from Eq. (20). Of course, the Aki and αi have to be learned, and Roth et al. (2019) use gradient descent to do so. We leave details to the original paper; briefly, they run in parallel a perturbed forward trajectory to estimate the LHS of Eq. (18) and then perform SGD on the squared difference between the LHS and RHS, giving gradients for Aki and αi.

KeRNL uses deterministic updates because it does not need explicit random variables. While the Bij(t) update is in closed form via Eq. (17), the updates for Aki and αi are numerical because of the need for SGD to train them to obey Eq. (18). Both its memory and computational complexities are 𝒪(n2).

3.6 Random-Feedback Online Learning (RFLO)

Coming from a computational neuroscience perspective, Murray (2019) developed a beautifully simple and biologically plausible learning rule for RNNs, which he calls Random-Feedback Online Learning (RFLO). He formulates the rule in terms of an eligibility trace Bij(t) that filters the non-zero immediate influence elements ϕ(hi(t))a^j(t-1) by the network inverse time constant α:


Then the approximate gradient is ultimately calculated11 1 As the “random feedback” part of the name suggests, Murray goes a step further in approximating c¯k(t) by random feedback weights á la Lillicrap et al., 2016, but we assume exact feedback in this paper for easier comparisons with other algorithms. as


By observing that


we see that RFLO is a special case of KeRNL, in which we fix Aki=δki, αi=α. Alternatively, and as hinted in the original paper, we can view RFLO as a special case of RTRL under the approximation Jkk(t)(1-α)δkk, because the RTRL update reduces to RFLO with Mkij(t)=δkiBij(t) containing Bij(t) along the diagonals:

Mkij(t) =kJkk(t)Mkij(t-1)+M¯kij(t)
=(1-α)Mkij(t-1)+αδkiϕ(hi(t))a^j(t-1). (22)

Fig. 2 illustrates how 𝐁(t) is contained in the influence matrix 𝐌(t). This algorithm’s update is deterministic and in closed form, with memory and computational complexity 𝒪(n2).

Figure 2: A visualization of the influence matrix and its 3 indices k, i, and j. In RFLO, the filtered immediate influences, stored in Bij(t), sparsely populate the influence matrix along the diagonals.

4 Future-facing algorithms

4.1 Backpropagation Through Time (BPTT)

For many applications, a recurrent network is unrolled only for some finite number of time steps, and backpropagation through time (BPTT) manifests as the computation of the sum L(t)/𝐰(s) over every st in the graph. This can be efficiently accomplished using

𝐜(t)=𝐜¯(t)+𝐉(t+1)𝐜(t+1) (23)

(see Eq. 7) to propagate credit assignment backwards. However, in our framework, where a network is run on an infinite-time horizon, there are two qualitatively different ways of unrolling the network. We call them “efficient” and “future-facing” BPTT.

4.1.1 Efficient backpropagation through time (E-BPTT)

For this method, we simply divide the graph into non-overlapping segments of truncation length T and perform BPTT between t-T and t as described above, using Eq. (23). It takes 𝒪(n2T) computation time to compute one gradient, but since this computation is only performed once every T time steps, the computation time is effectively 𝒪(n2), with memory requirement 𝒪(nT). A problem with this approach is that it does not treat all time points the same: an application of 𝐰 occurring near the end of the graph segment has less of its future influence accounted for than applications of 𝐰 occurring before it, as can be visualized in Fig. 3. And since any one gradient passed to the optimizer is a sum across both t and s, it is not an online algorithm by the framework we presented in §2. Therefore, for the purpose of comparing with online algorithms, we also show an alternative version of BPTT that calculates a future-facing gradient (up to truncation) /𝐰(t) for every t.

Figure 3: A visualization of various exact gradient methods. Each plot contains a lattice of points, representing derivatives L(t)/𝐰(s) for st, with gray boxes representing individual gradients passed to the optimizer. a) RTRL sums these derivatives into gradients for fixed t, using the PF relation (Eq. 5, §2) to efficiently derive successive gradients (blue arrow). b) F-BPTT sums these derivatives into gradients for fixed s by backpropagating through time (yellow arrows). c) E-BPTT creates a triangular gradient for non-overlapping subgraphs, using the FF relation (Eq. 7, §2) for efficient computation (red arrows). Here, the truncation horizon is T=4.

4.1.2 Future-facing backpropagation through time (F-BPTT)

In this version of BPTT, we keep a dynamic list of truncated credit assignment estimates 𝐜^(s) for times s=t-T,,t-1:


where each truncated credit assignment estimate includes the influences of 𝐚(s) only up to time t-1:


At current time t, every element 𝐜^(s) is extended by adding L(t)/𝐚(s), calculated by backpropagating from the current loss L(t), while the explicit credit assignment 𝐜¯(t) is appended to the front of the list. To compensate, the oldest credit assignment estimate 𝐜^(t-T) is removed and combined with the immediate influence to form a (truncated) gradient


which is passed to the optimizer to update the network. This algorithm is “online” in that it produces strictly future-facing gradients at each time step, albeit delayed by the truncation time T and requiring memory of the network states from t-T. Each update step requires 𝒪(n2T) computation, but since the update is performed at every time step, computation remains a factor of T more expensive than E-BPTT. Memory requirement is still 𝒪(nT). Fig. 3 illustrates the differences among these methods and RTRL, using a triangular lattice as a visualization tool. Each point in the lattice is one derivative L(t)/𝐰(s) with ts, and the points are grouped together into discrete gradients passed to the optimizer.

4.2 Decoupled Neural Interfaces (DNI)

Jaderberg et al. (2017) developed a framework for online learning by predicting credit assignment. Whereas PF algorithms face the problem of a large influence tensor Mkij(t) that needs a compressed representation, FF algorithms face the problem of incomplete information: at time t, it is impossible to calculate 𝐜(t) without access to future network variables. The approach of Decoupled Neural Interfaces (DNI) is to simply make a linear prediction of 𝐜(t) (Czarnecki et al., 2017) based on the current hidden state 𝐚(t) and the current labels 𝐲*(t):


where 𝐚~(t)=concat(𝐚(t),𝐲*(t),1)m, m=n+nout+1, and Ali are the components of a matrix 𝐀m×n, which parameterizes what the authors call the synthetic gradient function. The parameters Ali are trained to minimize the loss

LSG(t)=12la~l(t)Ali-ci(t)2 (24)

via gradient descent, similar to KeRNL’s treatment of Aki and αi (and we drop the time dependence of Ali for the same reason). Of course, this begs the question—the whole point is to avoid calculating 𝐜(t) explicitly, but calculating the error in Eq. (24) requires access to 𝐜(t). So the authors propose a “bootstrapping” technique analogous to the Bellman equation in Reinforcement Learnin (Sutton and Barto, 2018). If we take the FF relation we derived in Eq. (7)

𝐜(t)=𝐜¯(t)+𝐜(t+1)𝐉(t+1) (25)

and approximate the appearance of 𝐜(t+1) with the synthetic gradient estimate 𝐚~(t+1)𝐀, then Eq. (25) provides an estimate of ci(t) to use in Eq. (24). Then the update for 𝐀 can be written as

ΔAli-a~l(t)[la~l(t)Ali-(c¯i(t)+mla~l(t+1)AlmJmi(t+1))] (26)

with learning rate chosen as a hyperparameter. As in Eq. (8), the gradient is calculated by combining the estimated credit assignment for the i-th unit with the explicit influence by the ij-th parameter:


This algorithm is future facing because it ultimately estimates the effect of applying 𝐰 at current time t on future losses. Its updates are deterministic and numerical, because no explicit randomness is required, but the minimization problem over Ali implied by Eq. (24) is approximated by gradient descent rather than solved in closed form. It requires 𝒪(n2) memory for 𝐀 and 𝒪(n2) computation for the matrix-vector multiplications in Eq. (26).

4.2.1 Biological approximation to DNI

While many of the algorithms we have presented are biologically plausible in the abstract, i.e. temporally/spatially local and requiring no more than 𝒪(n2) memory, we have not yet discussed any explicit biological implementations. There are a handful of additional considerations for evaluating an algorithm with respect to biological plausibility:

  1. i

    Any equation describing synaptic strength changes (weight updates) must be local, i.e. any variables needed to update a synapse connecting the i-th and j-th units must be physically available to those units.

  2. ii

    Matrix-vector multiplication can be implemented by network-wide neural transmission, but input vectors must represent firing rates (e.g. post-activations 𝐚) and not membrane potentials (e.g. pre-activations 𝐡), since inter-neuron communication is mediated by spiking.

  3. iii

    Feedback weights used to calculate 𝐜¯ cannot be perfectly symmetric with 𝐖out, since there is no evidence for biological weight symmetry (see Lillicrap et al., 2016).

  4. iv

    Matrices (e.g. 𝐉 or 𝐀) must represent a set of synapses, whose strengths are determined by some local update.

With a few modifications, many of the presented algorithms can satisfy these requirements. We briefly illustrate one particular case with DNI, as shown in Marschall et al. (2019). To address (i), the result of the synthetic gradient operation la~l(t)Ali can be stored in an electrically isolated neural compartment, in a manner similar to biological implementations of feedforward backpropagation (Guerguiev et al., 2017; Sacramento et al., 2018), to allow for local updates to Wij. For (ii), simply pass the bootstrapped estimate of 𝐜¯(t+1) from Eq. (26) through the activation function ϕ so that it represents a neural firing rate. For (iii), one can use fixed, random feedback weights 𝐖fb instead of the output weights to calculate 𝐜¯(t), as in Lillicrap et al. (2016). And for (iv), one can train a set of weights 𝒥ij online to approximate the Jacobian by performing SGD on LJ(t)=ai(t)-j𝒥ijaj(t-1)2, which encodes the error of the linear prediction of the next network state by 𝒥ij. The update rule manifests as


essentially a “perceptron” learning rule, which is local and biologically realistic. Although this approximation brings no traditional computation speed benefits, it offers a plausible mechanism by which a neural circuit can access its own Jacobian for learning purposes. This technique could be applied to any other algorithm discussed in this paper. We refer to this altered version of DNI as DNI(b) in the experiments section.

5 Experiments

We run a number of experiments to empirically validate our categorizations and compare performance of the algorithms reviewed here, using two different synthetic tasks.

5.1 Setup

We implemented every algorithm presented here in a custom NumPy-based Python module.22 2 Link to public code repository to be included upon acceptance. In every simulation, we use gradient descent with a learning rate of 10-4, the fastest learning rate for which all algorithms are able to converge to a stable steady-state performance. We restrict ourselves to using a batch size of 1, because, in an online setting, the network must learn as data arrive in real time. Most algorithms demand additional configuration decisions and hyperparameter choices: the truncation horizon T (F-BPTT), the initial values of the tensors 𝐀 and 𝐁 (all approximations), the initial values of the learned timescales αi (KeRNL), the distribution from which 𝝂 is sampled for stochastic updates (UORO, KF-RTRL, R-KF-RTRL), and the learning rate for the numerical updates (KeRNL, DNI). For each algorithm, we independently optimize these choices by hand (see Appendix B for details).

We evaluate each algorithm’s ability to learn two different synthetic tasks: an additive dependencies task (Add) inspired by Pitis (2016) and a mimic target RNN task (Mimic). In both tasks, a stream of i.i.d. Bernoulli inputs 𝐱(t){0,1}nin is provided to the RNN. For Add, nin=nout=1. The label y*(t) has a baseline value of 0.5 that increases (or decreases) by 0.5 (or 0.25) if x=1 at t-t1 (or t-t2), for specified lags t1 and t2. The longer the lags of the dependencies, the more difficult the task. We choose t1=6 and t2=10. In the Mimic task, the labels 𝐲*(t)nout are determined by the outputs of a randomly generated, untrained target RNN that is fed the same input stream {𝐱(t):tt}. We use nin=nout=32 in Mimic, chosen so that learning 𝐖 is necessary for strong performance.

Figure 4: a) Cross-entropy loss for networks trained on Add task with α=1 for various algorithms. Lines are means over 20 random seeds (weight initialization and training set generation), and shaded regions represent ±1 S.E.M. Raw loss curves are first down-sampled by a factor of 10-4 (rectangular kernel) and then smoothed with a 10-time-step windowed running average. b) Same for α=0.5.

For each task, we consider a version on two different time scales: when the network is perfectly discrete (α=1, see Eq. 9) and when the network update has some time continuity (α=0.5). For the α=0.5 case, the tasks are stretched over time by a factor of 2 to compensate, and the dependencies are reduced to t1=3 and t2=5 in the Add task to keep the difficulty roughly the same.

5.2 Add task: result and analysis

Fig. 4 shows the performance of each learning algorithm on the Add task, in both α=1 and α=0.5 conditions, including a fixed 𝐖 algorithm that does not train 𝐖 at all but does train 𝐖out, as a baseline. As expected, since they compute exact gradients up to truncation, RTRL and F-BPTT perform best, although KF-RTRL is a sufficiently robust approximation to virtually tie with them. R-KF-RTRL and UORO perform similarly and worse than KF-RTRL does, as expected, since these approximations carry significantly more variance than KF-RTRL. However, in the α=0.5 condition, their performance is similar to that of KF-RTRL.

KeRNL is theoretically a stronger approximation of RTRL than RFLO, because of its ability to learn optimal Aki and αi whereas RFLO has fixed Aki=δki and αi=α. However, the numerical procedure for updating Aki and αi depends on several configuation/hyperparameter choices. Despite significant effort in exploring this space, we are not able to get KeRNL to perform better than RFLO, suggesting that the procedure for training Aki and αi does more harm than good. In the original paper, Roth et al. (2019) show promising results using RMSProp (Tieleman and Hinton, 2012) and batched training, so we suspect that the perturbation-based method for training Aki is simply too noisy for online learning.

DNI sits somewhere between the RFLO-KeRNL cluster and the rest, with its biologically realistic sibling DNI(b) performing slightly worse than DNI, as to be expected, since it is an approximation on top of an approximation. As with KeRNL, DNI’s numerical update of Ali introduces more hyperparameters and implementation choices, but there is a larger space of configurations in which the updates improve rather than hinder the algorithm’s performance.

Figure 5: Same as Fig. 4, for Mimic task with mean-squared-error loss.

5.3 Mimic task: result and analysis

For the Mimic task (Fig. 5), we see similar clustering of the algorithms but not in the same order. RTRL, F-BPTT, and KF-RTRL perform the best. UORO and R-KF-RTRL perform similarly to each other, but relatively worse than they did on Add. Conversely, DNI, DNI(b), KeRNL, and RFLO perform relatively better on this task than they did on Add. The information content that must be memorized for Add is relatively small (nin=1), but the time horizon is quite long. In Mimic, it is difficult to quantify the exact time horizon, but it is clear that the network must memorize much more information (nin=32). Thus perhaps UORO and R-KF-RTRL are effective at maintaining information over time, but the stochasticity in the updates places a limit on how much information can be retained.

5.4 Gradient similarity analysis

We conduct an in-depth investigation looking beyond task accuracy by directly comparing the gradients (or approximate gradients) produced by each algorithm. Fig. 6 shows how a given pair of algorithms align on a time-step-by-time-step basis for the Add and Mimic tasks. Each subplot is a histogram giving the distribution of normalized dot products

cos(θXY(t))=ΔX(t)𝐖ΔY(t)𝐖ΔX(t)𝐖ΔY(t)𝐖 (27)

for (flattened) weight updates ΔX(t)𝐖 and ΔY(t)𝐖 prescribed by algorithms X and Y, respectively, at time t.

Figure 6: a) Histograms of normalized gradient alignments for each pair of algorithms. Gradients are calculated during a simulation of 100k time steps of the Add task (same hyperparameters as in Figs. 4). Learning follows RTRL gradients, with other algorithms’ gradients passively computed for comparison. Mean alignment (dashed blue line) and 0 alignment reference (dashed black line) shown. b) Mean alignments of each approximate algorithm with RTRL and F-BPTT, color-coded by past facing (UORO, KF-RTRL, R-KF-RTRL, RFLO, KeRNL) vs. future facing (DNI). c) Same as (a), for Mimic task. d) Same as (b), for Mimic task.

Figs. 6 a,c show qualitatively similar trends:

  1. i

    As shown directly in Figs. 6b,d, PF algorithms (UORO, KF-RTRL, R-KF-RTRL, RFLO, KeRNL) align better with RTRL than with F-BPTT, and vice versa for FF algorithms (DNI).

  2. ii

    The deterministic PF algorithms (RFLO and KeRNL) align better with RTRL than the stochastic algorithms (UORO, KF-RTRL, and R-KF-RTRL) align with RTRL.

  3. iii

    RFLO and KeRNL align more strongly with each other than any other pair.

  4. iv

    UORO and R-KF-RTRL do not align strongly with any other algorithms, despite their ability to train the RNN effectively. UORO’s mean alignments with RTRL are 0.043 on Add and 0.084 on Mimic, while R-KF-RTRL’s mean alignments with RTRL are 0.050 on Add and 0.092 on Mimic (Fig. 6b,d), which are much lower than all other approximate algorithms, even those that perform worse on the task in some cases.

Observations (i)-(iii) validate our categorizations, as similarity according to the normalized alignment corresponds to similarity by the past-facing/future-facing, tensor structure, and stochastic/deterministic criteria. Observation (iv) is puzzling, as it shows that angular alignment with exact algorithms is not predictive of learning performance.

How are UORO and R-KF-RTRL able to learn at all if their gradients are almost orthogonal with RTRL on average? We address this question for both UORO and R-KF-RTRL by examining the joint distribution of the gradient’s alignment with RTRL and the gradient’s norm (Fig. 7). All 4 cases show a statistically significant positive linear correlation between the normalized alignment and the common log of the gradient norm. This observation may partially explain (iv), because larger weight updates occur when UORO happens to strongly align with RTRL. However, these correlations are fairly weak even if statistically significant, and we argue that better algorithm similarity metrics are needed to account for observed differences in performance.

Figure 7: The joint distribution of normalized alignments and gradient norms (log scale). a) UORO on Add task, b) UORO on Mimic task, c) R-KF-RTRL on Add task, d) R-KF-RTRL on Mimic task. Color intensity represents (smoothed) observed frequency, based on a sample of 100k time steps, with least-squares regression line in red. The estimated correlation coefficients r are all significant.

5.5 RFLO analysis

Among all approximate algorithms, RFLO stands out as having the simplest tensor structure and update rule, and it has been empirically demonstrated to be able to train RNNs on long-term dependencies. This is such a severe approximation of RTRL, yet it works so well in practice—and there is no clear understanding why. Although Murray (2019) goes into detail showing how loss decreases despite the random feedback used to approximately calculate 𝐜¯(t), he does not address the more basic mystery of how RFLO is able to learn despite the significant approximation 𝐉(t)(1-α)𝐈. In this section, we provide some intuition for how this simple learning rule is so successful and empirically validate our claims.

We hypothesize that, rather than learning dynamics that actively retain useful bits of the past like RTRL and BPTT, RFLO works by training what is essentially a high-capacity feedforward model to predict labels from the natural memory traces of previous inputs contained in the hidden state. This is reminiscient of reservoir computing (Lukoševičius and Jaeger, 2009). We illustrate this idea in the special case of a perfectly discrete network (α=1), where the learning rule still performs remarkably well despite Bij(t)=ϕ(hi(t))a^j(t-1) containing no network history. As Fig. 8a depicts, 𝐚(t-1) ultimately maps to 𝐲(t) via a single-hidden-layer feedforward network parameterized by 𝐖 and 𝐖out.

Figure 8: RFLO as a static multi-layer regression. a) One time step of recurrent dynamics combine with the output weights to form a single-hidden-layer feedforward network that is trained to predict 𝐲*(t) from 𝐚^(t-1) by the RFLO learning rule (ψ is softmax). b) Coefficient of determination r2 between 𝐚(t+Δt) and 𝐲*(t) as a function of the time shift Δt for different methods of generating 𝐖. Each trace is a different random initialization. The cyan dashed line shows the memory at Δt=-1, corresponding to the input 𝐚(t-1) of the network in (b). The black dashed lines indicate the lags of the input-output dependencies explicitly included in the task, corresponding to sudden jumps in information about 𝐲*(t) contained in the hidden state. c) Cross-entropy loss over learning by RFLO. The dashed lines indicate benchmarks for learning output statistics, the first dependency, and the second dependency, respectively (see Pitis, 2016 for details).

The RFLO learning rule in the discrete case corresponds exactly to training 𝐖 by backpropagation:

L(t)Wij=L(t)ai(t)ai(t)Wij=c¯i(t)ϕ(hi(t))a^j(t-1)=c¯i(t)Bij(t). (28)

While every learning algorithm additionally trains 𝐖out online to best map 𝐚(t) to 𝐲*(t), this purely linear model cannot perfectly capture the complex ways that information about past inputs 𝐱(t), tt implicit in 𝐚(t) relates to labels 𝐲*(t). Adding a hidden layer improves the ability of the network to predict 𝐲*(t) from whatever evidence of 𝐱(t) is naturally retained in 𝐚(t-1), analogous to how a single-hidden-layer feedforward network outperforms a simple softmax regression on MNIST (Deng, 2012).

To empirically validate our explanation, we first show that the strength of natural memory traces in the RNN depends on its recurrent weights. We measure this “memory” by running an untrained RNN with fixed weights forwards for 20k time steps of the Add task and calculating the r2 value of a linear regression of 𝐚(t+Δt) onto 𝐲*(t) for different values of the time shift Δt. The sudden jumps in information occur as Δt passes the time lags of the input-output dependencies explicitly included in the task (t1=6,t2=10), followed by a slow decay as the information relevant for predicting 𝐲*(t) gets corrupted by running the network forwards. The speed of this decay differs by the choice of 𝐖. As Fig. 8b shows, orthogonal and Gaussian 𝐖 seem to best preserve information over time, while symmetric and diagonal 𝐖 lose information quite rapidly, likely due to their real spectra. In separate simulations (Fig. 8c), we trained networks initialized in these ways using the RFLO rule, and only the networks initialized with orthogonal or Gaussian 𝐖 are able to learn at all with RFLO. This validates our hypothesis, that RFLO works by a static prediction of 𝐲*(t) based on evidence in 𝐚(t-1), because in cases where this evidence is absent (or at least weak) due to rapid decay, RFLO fails.

6 Discussion

We have presented a framework for conceptually unifying many recent advances in efficient online training of recurrent neural networks. These advances come from multiple perspectives: pure machine learning, biologically plausible machine learning, and computational neuroscience. We started by articulating a crucial distinction, (a) past facing vs. future facing, that divides algorithms by what exactly they calculate. We then presented a few other properties that characterize each new technique: (b) the tensor structure of the algorithm, (c) whether its update requires explicit randomness, and (d) whether its update can be derived in closed form from the relations (5)–(7) or must be approximated with SGD or some other numerical approach. Along the way, we clarified the relationship between some of the modern online approximations and exact methods. Further, we showed it’s possible to create new algorithms by identifying unexplored combinations of properties (a)-(d).

We empirically validated these ideas with synthetic tasks on which all algorithms could perform reasonably well in an online setting, despite using vanilla RNN architecture; gradient descent optimization with fixed learning rate; and no standard machine learning techniques such as gradient clipping, batch/layer normalization, L2 regularization, etc. We saw that training errors roughly cluster according to their categorical distinctions, across tasks and hyperparameter choices. But performance of these clusters differed across tasks: for example, UORO and R-KF-RTRL performed relatively well (poorly) on Add (Mimic), while for DNI and DNI(b) this effect was flipped. We speculate that this can be explained in terms of the task demands: the Add task has a long time horizon (10 time steps) of explicit dependencies for small inputs (nin=1), while the Mimic task requires more information to be memorized (nin=32) that decays with the target RNN’s forward pass (harder to measure, but likely shorter than 10 time steps). Thus perhaps UORO and R-KF-RTRL produce gradients with a limited amount of information that survives many updates, while DNI and DNI(b) have a larger information capacity but a limited time horizon. Future work should more systematically explore how these different task features interact with the different algorithmic approximations.

Following common practice, we used the pairwise vector alignments of the (approximate) gradients calculated by each algorithm as a way to analyze the precision of different approximations. This similarity turned out to reflect the natural clustering of the algorithms along the axes proposed here. In particular, past-facing approximations had stronger alignment with the exact past-facing gradients calculated by RTRL compared to the exact (up to truncation) future-facing gradients calculated by F-BPTT, and vice versa for the future-facing approximation, DNI. Reassuringly, KeRNL and RFLO, which have the same tensor structure, featured strong alignment.

Importantly, the angular alignment of the gradients did not account for task performance: UORO and R-KF-RTRL performed quite well despite their weak alignment with RTRL and BPTT, while KeRNL performed relatively poorly despite its strong alignment with RTRL and BPTT. Analyzing the magnitudes of the gradients partially explained this observation, as UORO and R-KF-RTRL aligned with RTRL more strongly when their gradients were larger in norm. However, this effect was subtle, so probably not enough to account for the performance differences. This exposes a limitation of current ways of analyzing this class of algorithms. There clearly is a need for better similarity metrics that go beyond time-point-wise gradient alignment and instead compare long-term learning trajectories.

Notably, it is the stochastic algorithms that have particularly weak alignment with RTRL—even KF-RTRL, yet its performance is indistinguishable from the exact algorithms. Averaged over many time steps of learning, corresponding to many samples of 𝝂, these stochastic methods do contain complete information about Mkij(t) (i.e. are unbiased), but at any one time point the alignment is heavily corrupted by the explicit randomness. In contrast, deterministic approximations, such as KeRNL, RFLO and DNI, may partially align with exact methods by construction, but their errors have no reason to average out, hence their inability to find the same minima as exact methods (Fig. 9). This may also explain why stochastic approximations do not align with each other despite their conceptual similarity.

Figure 9: Cartoon illustrating how alignment with RTRL and performance might dissociate. a) UORO’s noisy estimates of the true gradient are almost orthogonal with RTRL at each time point, but the errors average out over time and allow UORO to find the same solution. b) KeRNL aligns more strongly with RTRL at each time point, but errors do not average out, so KeRNL converges to a worse solution.

We chose the simplest possible setup for comparison, using basic RNN architecture and gradient descent, aiming to be as inclusive as possible of various algorithms. Of course, there are alternative online methods that do not naturally fall into this framework. Any online algorithm that exploits a particular RNN architecture necessarily cannot fit into our classification. For example, Ororbia et al. (2017, 2018) propose a specialized neural architecture (Temporal Neural Coding Network), whose learning algorithm (Discrepancy Reduction) is naturally local, due to the network structure. In contrast, the algorithms reviewed here were explicitly derived as approximations to nonlocal algorithms (RTRL and BPTT), and their locality ends up being more of a bug than a feature, as evidenced by their impaired performance. From a machine learning perspective, there are limits to general-purpose approximations of this kind, and future progress in online methods will likely come about by further exploring specialized algorithm-architecture pairs.

The same is true from a biological perspective. In the brain, cortical architecture and synaptic plasticity rules evolved together, while physical constraints dictate that plasticity is necessarily local. Still, the details of how local plasticity rules interact with neural circuits remain mysterious, and this is a current focus of research (Guerguiev et al., 2017; Sacramento et al., 2018; Lillicrap and Santoro, 2019). Exploring which architectures allow locality to manifest as a consequence, rather than a constraint, of learning is a potentially fruitful point of interaction between artificial intelligence and computational neuroscience.


We thank Guangyu Yang and James Murray for helpful conversations.


  • Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  • Bahdanau et al. (2016) Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End-to-end attention-based large vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4945–4949. IEEE, 2016.
  • Benzing et al. (2019) Frederik Benzing, Marcelo Matheus Gauy, Asier Mujika, Anders Martinsson, and Angelika Steger. Optimal kronecker-sum approximation of real time recurrent learning. arXiv preprint arXiv:1902.03993, 2019.
  • Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
  • Cho et al. (2015) Kyunghyun Cho, Aaron Courville, and Yoshua Bengio. Describing multimedia content using attention-based encoder-decoder networks. IEEE Transactions on Multimedia, 17(11):1875–1886, 2015.
  • Cooijmans and Martens (2019) Tim Cooijmans and James Martens. On the variance of unbiased online recurrent optimization. arXiv preprint arXiv:1902.02405, 2019.
  • Czarnecki et al. (2017) Wojciech Marian Czarnecki, Grzegorz Swirszcz, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 904–912. JMLR. org, 2017.
  • Deng (2012) Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  • Graves (2013) Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.
  • Graves et al. (2016) Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471, 2016.
  • Guerguiev et al. (2017) Jordan Guerguiev, Timothy P Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites. ELife, 6:e22901, 2017.
  • Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  • Jaderberg et al. (2017) Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1627–1635. JMLR. org, 2017.
  • Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  • Lillicrap and Santoro (2019) Timothy P Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current Opinion in Neurobiology, 55:82 – 89, 2019. ISSN 0959-4388. doi: https://doi.org/10.1016/j.conb.2019.01.011. URL http://www.sciencedirect.com/science/article/pii/S0959438818302009. Machine Learning, Big Data, and Neuroscience.
  • Lillicrap et al. (2016) Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7:13276, 2016.
  • Lukoševičius and Jaeger (2009) Mantas Lukoševičius and Herbert Jaeger. Reservoir computing approaches to recurrent neural network training. Computer Science Review, 3(3):127–149, 2009.
  • Marschall et al. (2019) Owen Marschall, Kyunghyun Cho, and Cristina Savin. Using local plasticity rules to train recurrent neural networks. In Computational and Systems Neuroscience (CoSyNe), number II-90, Lisbon, Portugal, Feb 2019.
  • Mikolov et al. (2010) Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
  • Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
  • Mujika et al. (2018) Asier Mujika, Florian Meier, and Angelika Steger. Approximating real-time recurrent learning with random kronecker factors. In Advances in Neural Information Processing Systems, pages 6594–6603, 2018.
  • Murray (2019) James M Murray. Local online learning in recurrent networks with random feedback. eLife, 8:e43299, 2019.
  • Ororbia et al. (2018) Alexander Ororbia, Ankur Mali, C Lee Giles, and Daniel Kifer. Online learning of recurrent neural architectures by locally aligning distributed representations. arXiv preprint arXiv:1810.07411, 2018.
  • Ororbia et al. (2017) II Ororbia, G Alexander, Patrick Haffner, David Reitter, and C Lee Giles. Learning to adapt by minimizing discrepancy. arXiv preprint arXiv:1711.11542, 2017.
  • Pitis (2016) Silviu Pitis. Recurrent neural networks in tensorflow 1. r2rt.com/recurrent-neural-networks-in-tensorflow-i, 2016. Accessed: 2018-11-13.
  • Roth et al. (2019) Christopher Roth, Ingmar Kanitscheider, and Ila Fiete. Kernel RNN learning (keRNL). In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=ryGfnoC5KQ.
  • Sacramento et al. (2018) João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems, pages 8721–8732, 2018.
  • Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
  • Tallec and Ollivier (2017) Corentin Tallec and Yann Ollivier. Unbiased online recurrent optimization. arXiv preprint arXiv:1702.05043, 2017.
  • Tieleman and Hinton (2012) T. Tieleman and G. Hinton. Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
  • Werbos et al. (1990) Paul J Werbos et al. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560, 1990.
  • Williams and Zipser (1989) Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270–280, 1989.

Appendix A.

Lemma for generating rank-1 unbiased estimates

For completeness, we state the Lemma from Tallec and Ollivier (2017) in components notation. Given a decomposition of a matrix 𝐌n×m into r rank-1 components

Mij=k=1rAikBkj, (29)

a vector of i.i.d. random variables 𝝂r with 𝔼[νk]=1, 𝔼[νkνk]=δkk, and a list of r positive constants ρk>0, then

M~ij=(k=1rρkνkAik)(k=1rρk-1νkBkj) (30)

is a rank-1, unbiased estimate of Mij over the choice of 𝝂.

Appendix B.

Implementation details

For reproducibility, we describe in fuller detail our implementation configurations for each simulation. Table 2 shows hyperparameter/configuration choices that apply across all algorithms. Table 3 shows the algorithm-specific hyperparameter choices we made for each task. In Table 2, we reference sub-matrices of 𝐖=[𝐰rec,𝐰in,𝐛rec] and 𝐖out=[𝐰out,𝐛rec], since they are initialized differently.

hyperparameter value explanation
learning rate 10-4 learning rate for SGD w.r.t. 𝐖 and 𝐖out
n 32 number of hidden units in the network
ϕ tanh nonlinearity used in RNN forward dynamics
init. 𝐰in 𝒩(0,1/nin) initial value for input weights
init. 𝐰rec rand. orth. initial value for recurrent weights
init. 𝐛rec 0 initial value for recurrent bias
init. 𝐰out 𝒩(0,1/n) initial value for output weights
init. 𝐛out 0 initial value for output bias
init. 𝐖FB 𝒩(0,1/nout) value for fixed feedback weights used in DNI(b)
init. 𝐛targ.rec 𝒩(0,0.1) initial value for target recurrent bias in Mimic
init. 𝐛targ.out 𝒩(0,0.1) initial value for target output bias in Mimic
Table 2: Default hyperparameter choices for the RNN independent of learning algorithm.
algorithm initial values 𝝂 dist. LR pert. σ T
UORO Ak(0)𝒩(0,1),Bij(0)𝒩(0,1) unif. {-1,1}
KF-RTRL Aj(0)𝒩(0,1),Bki(0)𝒩(0,1/n) unif. {-1,1}
R-KF-RTRL Ai(0)𝒩(0,1),Bkj(0)𝒩(0,1/n) unif. {-1,1}
KeRNL Aki(0)=δki,Bij(0)=0,αi(0)=0.8 5 10-3
DNI Ali(0)𝒩(0,1/n) 10-3
DNI(b) Ali(0)𝒩(0,1/n),𝒥ij(0)=Wijrec 10-3
Table 3: Hyperparameter choices specific to individual algorithms.

Some miscellaneous implementation details below:

  • For the Add task in the α=1 condition, we changed the DNI/DNI(b) learning rate to 5×10-2 for Ali and 10-2 for 𝒥ij (DNI(b)). In other cases, the learning rates for Ali and 𝒥ij are identical.

  • There are two appearances of the synthetic gradient weights Ali in Eq. (26). Although we wrote them as one matrix 𝐀 for brevity, in implementation we actually keep two separate values, 𝐀 and 𝐀*, the latter of which we use for for the right-hand appearance Alm (specifically to calculate the bootstrapped estimate of the SG training label). We update 𝐀 every time step but keep 𝐀* constant, replacing it with the latest value of 𝐀 only once per τ time steps. This integer τ introduces another hyperparameter, which we choose to be 5. (Inspired by an analogous technique used in deep Q-learning from Mnih et al., 2015.)

  • In the original paper, Roth et al. (2019) use (1-exp(-γi)) rather than αi as a temporal filter for Bij(t). We made this change so that αi makes sense in terms of the α in the forward dynamics of the network and RFLO. Of course, these are equivalent via γi=-log(1-αi), but the gradient w.r.t. αi must be rescaled by a factor of 1/(1-αi) to compensate.

  • For KeRNL, there is a choice for how to update the eligibility trace (Eq. 17): one can scale the right-hand term ϕ(hi(t))a^j(t-1) by either the learned timescale αi or the RNN timescale α. We chose the latter because it has stronger empirical performance and it theoretically recovers the RTRL equation under the approximating assumptions about 𝐀.

  • Perturbations for calculating gradients for Aki and αi in KeRNL are sampled i.i.d. ζi𝒩(0,σ).

  • In our implementation of the Add task, we use nin=nout=2 for a “one-hot” representation of the input x(t){0,1} and label y*(t){0.25,0.5,0.75,1}, such that 𝐱(t)=[x(t),1-x(t)] and 𝐲*(t)=[y(t),1-y(t)].

  • In our implementation of Mimic, the target RNN was initialized in the same way as the RNNs we train, with the exception of the recurrent and output biases (see Table 2).