We present a framework for compactly summarizing many recent results inefficient and/or biologically plausible online training of recurrent neuralnetworks (RNN). The framework organizes algorithms according to severalcriteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs.deterministic, and (d) closed form vs. numerical. These axes reveal latentconceptual connections among several recent advances in online learning.Furthermore, we provide novel mathematical intuitions for their degree ofsuccess. Testing various algorithms on two synthetic tasks shows thatperformances cluster according to our criteria. Although a similar clusteringis also observed for gradient alignment, alignment with exact methods does notalone explain ultimate performance, especially for stochastic algorithms. Thissuggests the need for better comparison metrics.
Quick Read (beta)
A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks
We present a framework for compactly summarizing many recent results in efficient and/or biologically plausible online training of recurrent neural networks (RNN). The framework organizes algorithms according to several criteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs. deterministic, and (d) closed form vs. numerical. These axes reveal latent conceptual connections among several recent advances in online learning. Furthermore, we provide novel mathematical intuitions for their degree of success. Testing various algorithms on two synthetic tasks shows that performances cluster according to our criteria. Although a similar clustering is also observed for gradient alignment, alignment with exact methods does not alone explain ultimate performance, especially for stochastic algorithms. This suggests the need for better comparison metrics.
Keywords: real-time recurrent learning, backpropagation through time, approximation, biologically plausible learning, local, online
Training recurrent neural networks (RNN) to learn sequence data is traditionally done with stochastic gradient descent (SGD), using the backpropagation through time algorithm (BPTT, Werbos et al., 1990) to calculate the gradient. This requires “unrolling” the network over some range of time steps and performing backpropagation as though the network were feedforward under the constraint of sharing parameters across time steps (“layers”). BPTT’s success in a wide range of applications (Mikolov et al., 2010; Graves, 2013; Bahdanau et al., 2016, 2014; Cho et al., 2015; Graves et al., 2016) has made it the industry standard; however, there exist alternative online algorithms for training RNNs. These compute gradients in real time as the network runs forward, without explicitly referencing past activity or averaging over batches of data. There are two reasons for considering online alternatives to BPTT. One is practical: computational costs do not scale with . The other is conceptual: human brains are able to learn long-term dependencies without explicitly memorizing all past brain states, and understanding online learning is a key step in the larger project of understanding human learning.
The classic online learning algorithm is real-time recurrent learning (RTRL, Williams and Zipser, 1989), which is equivalent to BPTT in the limit of a small learning rate (Murray, 2019). RTRL recursively updates the total derivative of the hidden state with respect to the parameters, eliminating the need to reference past activity but introducing an order memory requirement. In practice, this is often more computationally demanding than BPTT (order ), hence not frequently used in applications. Nor is RTRL at face value a good model of biological learning, for the same reason: no known biological mechanism exists to store—let alone manipulate—a float for each synapse-neuron pair. Thus RTRL and online learning more broadly have remained relatively obscure footnotes to both the deep learning revolution itself and its impact on computational neuroscience.
Recent advances in recurrent network architectures have brought the issue of online learning back into the spotlight. While vanishing gradients used to significantly limit the extent of the temporal dependencies that an RNN could learn, new architectures like LSTMs (Hochreiter and Schmidhuber, 1997) and GRUs (Cho et al., 2014) have dramatically expanded this learnable time horizon. Unfortunately, taking advantage of this capacity requires an equally dramatic expansion in computational resources, if using BPTT. This has led to an explosion of novel online learning algorithms (Tallec and Ollivier, 2017; Mujika et al., 2018; Roth et al., 2019; Murray, 2019; Jaderberg et al., 2017) which aim to improve on the efficiency of RTRL, in many cases using update rules that might be implemented by a biological circuit.
The sheer number and variety of these approaches pose challenges for both theory and practice. It is not always completely clear what makes various algorithms different from one another, how they are conceptually related, or even why they might work in the first place. There is a pressing need in the field for a cohesive framework for describing and comparing online methods. Here we aim to provide a thorough overview of modern online algorithms for training RNNs, in a way that provides a clearer understanding of the mathematical structure underlying all these different approaches. Our framework organizes the existing literature along several axes that encode meaningful conceptual distinctions:
Past facing vs. future facing
The tensor structure of the algorithm
Stochastic vs. deterministic update
Closed form vs. numerical solution for update
These axes will be explained in detail later, but briefly: the past vs. future axis is a root distinction that divides algorithms by the type of gradient they calculate, while the other three describe their representations and update principles. Table 1 contains (to our knowledge) all recently published online learning algorithms for RNNs, categorized according to these criteria. We can already see that many combinations of these characteristics manifest in the literature, suggesting that new algorithms could be developed by mixing and matching properties. (We provide a concrete example of this in §3.4.)
Here we describe each algorithm in unified notation that makes clear their classification by these criteria. In the process, we generate novel intuitions about why different approximations can be successful and discuss some of the finer points of their biological plausibility. Finally, we simulate each algorithm on a common set of synthetic tasks with vanilla RNN architecture for simplicity. We compare performance and analyze gradient alignments to see to what extent their empirical similarity is predicted by their similarity according to our framework. Algorithm performance roughly clusters according to criteria (a)-(d) across tasks, lending credence to our approach. Curiously, gradient alignment with exact methods (RTRL and BPTT) does not predict performance, despite its ubiquity as a tool for analyzing approximate learning algorithms.
2 Past- and future-facing perspectives of online learning
Before we dive into the details of these algorithms, we first articulate what we mean by past- and future-facing, related to the “reverse/forward accumulation” distinction concurrently described by Cooijmans and Martens (2019). Consider a recurrent neural network that contains, at each time step , a state . This state is updated via a function , which is parameterized by a flattened vector of parameters . Here counts the total number of input dimensions, including the recurrent inputs , task inputs , and an additional input clamped to (to represent bias). For some initial state , defines the network dynamics by
At each time step an output is computed by another function , parameterized by . We will typically choose an affine-softmax readout for , with output weights/bias . A loss function calculates an instantaneous loss , quantifying to what degree the predicted output matches the target output .
The goal is to train the network by gradient descent (or other gradient-based optimizers such as ADAM from Kingma and Ba, 2014) on the total loss w.r.t. the parameters and . It is natural to learn online, because only information at present time is required to calculate the gradient . So the heart of the problem is to calculate .
The parameter is applied via at every time step, and we denote a particular application of at time as . Of course, a recurrent system is constrained to share parameters across time steps, so a perturbation is effectively a perturbation across all applications , i.e., . In principle, each application of the parameters affects all future losses , . The core of any recurrent learning algorithm is to estimate the influence of one parameter application on one loss , since these individual terms are necessary and sufficient to define the global gradient
This raises the question of how to sum these components to produce individual gradients to pass to the optimizer. In truncated BPTT, one unrolls the graph over some range of time steps and sums for all in that range with (see §4.1.1). This does not qualify as an “online” learning rule, because it requires two independent time indices—at most one can represent “real time” leaving the other to represent the future or the past. If we can account for one of the summations via dynamic updates, then the algorithm is online or temporally local, i.e. not requiring explicit reference to the past or future. As depicted in Fig. 1, there are two possibilities. If from Eq. (1) corresponds to real time, then the gradient passed to the optimizer is
In this case, we say learning is past facing, because the gradient is a sum of the influences of past applications of on the current loss. On the other hand, if from Eq. (1) represents real time, then the gradient passed to the optimizer is
Here we say learning is future facing, because the gradient is a sum of influences by the current application of on future losses.
2.1 Past-facing online learning algorithms
Here we derive a fundamental relation leveraged by past-facing (PF) online algorithms. Let index real time, and define the influence matrix , where and are respectively the number of hidden units and the number of parameters defining . tracks the derivatives of the current state with respect to each parameter :
Let’s rewrite Eq. (4) with matrix notation and unpack it by one time step:
A simple recursive formula emerges, wherein the influence matrix is updated by multiplying its current value by the Jacobian of the network and then adding the immediate influence . To compute the gradient that ultimately gets passed to the optimizer, we simply use the chain rule over the current hidden state :
where the immediate credit assignment vector is defined to be and is calculated by backpropagating the error through the derivative of the output function (or approximated by Feedback Alignment, see Lillicrap et al., 2016). In the end, we compute a derivative in Eq. (6) that is implicitly a sum over the many terms of Eq. (2), using formulae that depend explicitly only on times and . For this reason, such a learning algorithm is online, and it is past facing because the gradient computation is of the form in Eq. (2).
2.2 Future-facing online learning algorithms
Here we show a symmetric relation for future-facing (FF) online algorithms. The credit assignment vector is a row vector defined as the gradient of the loss with respect to the hidden state . It plays a role analogous to and has a recursive update similar to Eq. (5):
As in the PF case, the gradient is ultimately calculated using the chain rule over :
The recursive relations for PF and FF algorithms are of identical form given the following changes: (1) swap the roles of and , (2) swap the roles of and , and (3) flip the direction of all derivatives. This clarifies the fundamental trade-off between the PF and FF approaches to online learning. On the one hand, memory requirements favor FF because is a scalar while is a matrix. On the other, only PF can truly be run online, because the time direction of the update in FF is opposite the forward pass. Thus, efficient PF algorithms must compress , while efficient FF algorithms must predict .
3 Past-facing algorithms
3.1 Real-Time Recurrent Learning
The Real-Time Recurrent Learning (RTRL, Williams and Zipser, 1989) algorithm directly applies Eqs. (5) and (6) as written. We call the application of Eq. (5) the “update” to the learning algorithm, which is deterministic and in closed form. Implementing Eq. (5) requires storing floats in and performing multiplications in , which is neither especially efficient nor biologically plausible. However, several efficient (and in some cases, biologically plausible) online learning algorithms have recently been developed, including Unbiased Online Recurrent Optimization (UORO; Tallec and Ollivier, 2017; §3.2), Kronecker-Factored RTRL (KF-RTRL; Mujika et al., 2018; §3.3), Kernel RNN Learning (KeRNL; Roth et al., 2019; §3.5), and Random-Feedback Online Learning (RFLO; Murray, 2019; §3.6). We claim that these learning algorithms, whether explicitly derived as such or not, are all implicitly approximations to RTRL, each a special case of a general class of techniques for compressing . In the following section, we clarify how each of these learning algorithms fits into this broad structure.
3.1.1 Approximations to RTRL
To concretely illuminate these ideas, we will work with a special case of , a time-continuous vanilla RNN:
where , , is some point-wise nonlinearity (e.g. ), and is the network’s inverse time constant. The trainable parameters are folded via the indexing into the weight matrix , whose columns hold the recurrent weights, the input weights, and a bias. By reshaping into its natural matrix form , we can write the influence matrix as an order-3 influence tensor
Thus specifies the effect on the -th unit of perturbing the direct connection from the -th unit to the -th unit. The immediate influence can also be written as a tensor. By differentiating Eq. (9), we see it takes the sparse form
because can affect the -th unit directly only if . Many approximations of RTRL involve a decomposition of into a product of lower-order tensors. For example, UORO represents by an outer product , which has a memory requirement of only . Similarly, KF-RTRL uses a Kronecker-product decomposition . We can generalize these cases into a set of six possible decompositions of into products of lower-order tensors and :
Each such decomposition has a memory requirement of . Of course, it is not sufficient to write down an idealized decomposition for a particular time point; there must exist some efficient way to update the decomposition as the network runs forwards. We now go through each algorithm and show the mathematical techniques used to derive update equations and categorize them by the criteria outlined in Table 1.
3.2 Unbiased Online Recurrent Optimization (UORO)
Tallec and Ollivier (2017) discovered a technique for approximating as an outer product , where and . The authors proved a crucial lemma (see Appendix A or Tallec and Ollivier, 2017) that gives, in closed form, an unbiased rank-1 estimate of a given matrix over the choice of a random vector with and . They leverage this result to derive a closed-form update rule for and at each time step, without ever having to explicitly (and expensively) calculate . We present an equivalent formulation in terms of tensor components, i.e.,
where represents the “rolled-up” components of , as in w.r.t. . Intuitively, the -th component of the influence matrix is constrained to be the product of the -th unit’s “sensitivity” and the -th parameter’s “efficacy” . Eqs. (10) and (11) show the form of the update and why it is unbiased over , respectively:
The cross terms vanish in expectation because . Thus, by induction over , the estimate of remains unbiased at every time step. The constants are chosen at each time step to minimize total variance of the estimate by balancing the norms of the cross terms. This algorithm’s update is stochastic due to its reliance on the random vector , but it is in closed form because it has an explicit update formula (Eq. 10). Both its memory and computational complexity are .
3.3 Kronecker-Factored RTRL (KF-RTRL)
Mujika et al. (2018) leverage the same lemma as in UORO, but using a decomposition of in terms of a Kronecker product , where now and . This decomposition is more natural, because the immediate influence factors exactly as a Kronecker product for vanilla RNNs, where . To derive the update rule for UORO, one must first generate a rank-1 estimate of as an intermediate step, introducing more variance, but in KF-RTRL, this step is unnecessary. In terms of components, the compression takes the form
As in UORO, the cross terms vanish in expectation, and the estimate is unbiased by induction over . This algorithm’s updates are also stochastic and in closed form. Its memory complexity is , but its computation time is because of the matrix-matrix product in Eq. (13).
3.4 Reverse KF-RTRL (R-KF-RTRL)
Our exploration of the space of different approximations naturally raises a question: is an approximation of the form
also possible? We refer to this method as “Reverse” KF-RTRL (R-KF-RTRL) because, in matrix notation, this would be formulated as , where and . We propose the following update for and in terms of a random vector :
Eq. (16) shows that this estimate is unbiased, using updates that are stochastic and in closed form, like its sibling algorithms. Its memory and computational complexity are and , respectively. R-KF-RTRL is actually more similar to UORO than KF-RTRL, because does not naturally factor like Eq. (14), introducing more variance. Worse, it has the computational complexity of KF-RTRL due to the matrix-matrix multiplication in Eq. (15). KF-RTRL stands out as the most effective of these 3 algorithms, because it estimates with the lowest variance due to its natural decomposition structure. (See Mujika et al., 2018 for variance calculations.)
3.4.1 Optimal Kronecker-Sum Approximation (OK)
We briefly mention an extension of KF-RTRL by Benzing et al. (2019), where the influence matrix is approximated not by 1 but rather a sum of Kronecker products, or, in components
On the RTRL update, the index of is propagated forward by the Jacobian, and then the immediate influence—itself a Kronecker product—is added. Now is approximated by Kronecker products
but the authors developed a technique to optimally reduce this sum back to Kronecker products, keeping the memory complexity and computational complexity constant. This update is stochastic because it requires explicit randomness in the flavor of the above algorithms, and it is numerical because there is no closed form solution to the update. We leave the details to the original paper.
3.5 Kernel RNN Learning (KeRNL)
Roth et al. (2019) developed a learning algorithm for RNNs that is essentially a compression of the influence matrix of the form . We will show that this algorithm is also an implicit approximation of RTRL, although the update rules are fundamentally different than those for UORO, KF-RTRL and R-KF-RTRL. The eligibility trace updates by temporally filtering the immediate influences with unit-specific, learnable timescales :
The sensitivity matrix is chosen to approximate the multi-step Jacobian with help from the learned timescales:
We will describe how is learned later, but for now we assume this approximation holds and use it to show how the KeRNL update is equivalent to that of RTRL. We have dropped the explicit time-dependence from , because it updates too slowly for Eq. (18) to be specific to any one time point. If we unpack this approximation by one time step, we uncover the consistency relation
Then the eligibility trace update effectively implements the RTRL update, assuming inductively that is well approximated by :
In Eq. (21), we use each of the special cases from Eq. (20). Of course, the and have to be learned, and Roth et al. (2019) use gradient descent to do so. We leave details to the original paper; briefly, they run in parallel a perturbed forward trajectory to estimate the LHS of Eq. (18) and then perform SGD on the squared difference between the LHS and RHS, giving gradients for and .
3.6 Random-Feedback Online Learning (RFLO)
Coming from a computational neuroscience perspective, Murray (2019) developed a beautifully simple and biologically plausible learning rule for RNNs, which he calls Random-Feedback Online Learning (RFLO). He formulates the rule in terms of an eligibility trace that filters the non-zero immediate influence elements by the network inverse time constant :
Then the approximate gradient is ultimately calculated11 1 As the “random feedback” part of the name suggests, Murray goes a step further in approximating by random feedback weights á la Lillicrap et al., 2016, but we assume exact feedback in this paper for easier comparisons with other algorithms. as
By observing that
we see that RFLO is a special case of KeRNL, in which we fix , . Alternatively, and as hinted in the original paper, we can view RFLO as a special case of RTRL under the approximation , because the RTRL update reduces to RFLO with containing along the diagonals:
Fig. 2 illustrates how is contained in the influence matrix . This algorithm’s update is deterministic and in closed form, with memory and computational complexity .
4 Future-facing algorithms
4.1 Backpropagation Through Time (BPTT)
For many applications, a recurrent network is unrolled only for some finite number of time steps, and backpropagation through time (BPTT) manifests as the computation of the sum over every in the graph. This can be efficiently accomplished using
(see Eq. 7) to propagate credit assignment backwards. However, in our framework, where a network is run on an infinite-time horizon, there are two qualitatively different ways of unrolling the network. We call them “efficient” and “future-facing” BPTT.
4.1.1 Efficient backpropagation through time (E-BPTT)
For this method, we simply divide the graph into non-overlapping segments of truncation length and perform BPTT between and as described above, using Eq. (23). It takes computation time to compute one gradient, but since this computation is only performed once every time steps, the computation time is effectively , with memory requirement . A problem with this approach is that it does not treat all time points the same: an application of occurring near the end of the graph segment has less of its future influence accounted for than applications of occurring before it, as can be visualized in Fig. 3. And since any one gradient passed to the optimizer is a sum across both and , it is not an online algorithm by the framework we presented in §2. Therefore, for the purpose of comparing with online algorithms, we also show an alternative version of BPTT that calculates a future-facing gradient (up to truncation) for every .
4.1.2 Future-facing backpropagation through time (F-BPTT)
In this version of BPTT, we keep a dynamic list of truncated credit assignment estimates for times :
where each truncated credit assignment estimate includes the influences of only up to time :
At current time , every element is extended by adding , calculated by backpropagating from the current loss , while the explicit credit assignment is appended to the front of the list. To compensate, the oldest credit assignment estimate is removed and combined with the immediate influence to form a (truncated) gradient
which is passed to the optimizer to update the network. This algorithm is “online” in that it produces strictly future-facing gradients at each time step, albeit delayed by the truncation time and requiring memory of the network states from . Each update step requires computation, but since the update is performed at every time step, computation remains a factor of more expensive than E-BPTT. Memory requirement is still . Fig. 3 illustrates the differences among these methods and RTRL, using a triangular lattice as a visualization tool. Each point in the lattice is one derivative with , and the points are grouped together into discrete gradients passed to the optimizer.
4.2 Decoupled Neural Interfaces (DNI)
Jaderberg et al. (2017) developed a framework for online learning by predicting credit assignment. Whereas PF algorithms face the problem of a large influence tensor that needs a compressed representation, FF algorithms face the problem of incomplete information: at time , it is impossible to calculate without access to future network variables. The approach of Decoupled Neural Interfaces (DNI) is to simply make a linear prediction of (Czarnecki et al., 2017) based on the current hidden state and the current labels :
where , , and are the components of a matrix , which parameterizes what the authors call the synthetic gradient function. The parameters are trained to minimize the loss
via gradient descent, similar to KeRNL’s treatment of and (and we drop the time dependence of for the same reason). Of course, this begs the question—the whole point is to avoid calculating explicitly, but calculating the error in Eq. (24) requires access to . So the authors propose a “bootstrapping” technique analogous to the Bellman equation in Reinforcement Learnin (Sutton and Barto, 2018). If we take the FF relation we derived in Eq. (7)
with learning rate chosen as a hyperparameter. As in Eq. (8), the gradient is calculated by combining the estimated credit assignment for the -th unit with the explicit influence by the -th parameter:
This algorithm is future facing because it ultimately estimates the effect of applying at current time on future losses. Its updates are deterministic and numerical, because no explicit randomness is required, but the minimization problem over implied by Eq. (24) is approximated by gradient descent rather than solved in closed form. It requires memory for and computation for the matrix-vector multiplications in Eq. (26).
4.2.1 Biological approximation to DNI
While many of the algorithms we have presented are biologically plausible in the abstract, i.e. temporally/spatially local and requiring no more than memory, we have not yet discussed any explicit biological implementations. There are a handful of additional considerations for evaluating an algorithm with respect to biological plausibility:
Any equation describing synaptic strength changes (weight updates) must be local, i.e. any variables needed to update a synapse connecting the -th and -th units must be physically available to those units.
Matrix-vector multiplication can be implemented by network-wide neural transmission, but input vectors must represent firing rates (e.g. post-activations ) and not membrane potentials (e.g. pre-activations ), since inter-neuron communication is mediated by spiking.
Feedback weights used to calculate cannot be perfectly symmetric with , since there is no evidence for biological weight symmetry (see Lillicrap et al., 2016).
Matrices (e.g. or ) must represent a set of synapses, whose strengths are determined by some local update.
With a few modifications, many of the presented algorithms can satisfy these requirements. We briefly illustrate one particular case with DNI, as shown in Marschall et al. (2019). To address (i), the result of the synthetic gradient operation can be stored in an electrically isolated neural compartment, in a manner similar to biological implementations of feedforward backpropagation (Guerguiev et al., 2017; Sacramento et al., 2018), to allow for local updates to . For (ii), simply pass the bootstrapped estimate of from Eq. (26) through the activation function so that it represents a neural firing rate. For (iii), one can use fixed, random feedback weights instead of the output weights to calculate , as in Lillicrap et al. (2016). And for (iv), one can train a set of weights online to approximate the Jacobian by performing SGD on , which encodes the error of the linear prediction of the next network state by . The update rule manifests as
essentially a “perceptron” learning rule, which is local and biologically realistic. Although this approximation brings no traditional computation speed benefits, it offers a plausible mechanism by which a neural circuit can access its own Jacobian for learning purposes. This technique could be applied to any other algorithm discussed in this paper. We refer to this altered version of DNI as DNI(b) in the experiments section.
We run a number of experiments to empirically validate our categorizations and compare performance of the algorithms reviewed here, using two different synthetic tasks.
We implemented every algorithm presented here in a custom NumPy-based Python module.22 2 Link to public code repository to be included upon acceptance. In every simulation, we use gradient descent with a learning rate of , the fastest learning rate for which all algorithms are able to converge to a stable steady-state performance. We restrict ourselves to using a batch size of 1, because, in an online setting, the network must learn as data arrive in real time. Most algorithms demand additional configuration decisions and hyperparameter choices: the truncation horizon (F-BPTT), the initial values of the tensors and (all approximations), the initial values of the learned timescales (KeRNL), the distribution from which is sampled for stochastic updates (UORO, KF-RTRL, R-KF-RTRL), and the learning rate for the numerical updates (KeRNL, DNI). For each algorithm, we independently optimize these choices by hand (see Appendix B for details).
We evaluate each algorithm’s ability to learn two different synthetic tasks: an additive dependencies task (Add) inspired by Pitis (2016) and a mimic target RNN task (Mimic). In both tasks, a stream of i.i.d. Bernoulli inputs is provided to the RNN. For Add, . The label has a baseline value of 0.5 that increases (or decreases) by 0.5 (or 0.25) if at (or ), for specified lags and . The longer the lags of the dependencies, the more difficult the task. We choose and . In the Mimic task, the labels are determined by the outputs of a randomly generated, untrained target RNN that is fed the same input stream . We use in Mimic, chosen so that learning is necessary for strong performance.
For each task, we consider a version on two different time scales: when the network is perfectly discrete (, see Eq. 9) and when the network update has some time continuity (). For the case, the tasks are stretched over time by a factor of 2 to compensate, and the dependencies are reduced to and in the Add task to keep the difficulty roughly the same.
5.2 Add task: result and analysis
Fig. 4 shows the performance of each learning algorithm on the Add task, in both and conditions, including a fixed algorithm that does not train at all but does train , as a baseline. As expected, since they compute exact gradients up to truncation, RTRL and F-BPTT perform best, although KF-RTRL is a sufficiently robust approximation to virtually tie with them. R-KF-RTRL and UORO perform similarly and worse than KF-RTRL does, as expected, since these approximations carry significantly more variance than KF-RTRL. However, in the condition, their performance is similar to that of KF-RTRL.
KeRNL is theoretically a stronger approximation of RTRL than RFLO, because of its ability to learn optimal and whereas RFLO has fixed and . However, the numerical procedure for updating and depends on several configuation/hyperparameter choices. Despite significant effort in exploring this space, we are not able to get KeRNL to perform better than RFLO, suggesting that the procedure for training and does more harm than good. In the original paper, Roth et al. (2019) show promising results using RMSProp (Tieleman and Hinton, 2012) and batched training, so we suspect that the perturbation-based method for training is simply too noisy for online learning.
DNI sits somewhere between the RFLO-KeRNL cluster and the rest, with its biologically realistic sibling DNI(b) performing slightly worse than DNI, as to be expected, since it is an approximation on top of an approximation. As with KeRNL, DNI’s numerical update of introduces more hyperparameters and implementation choices, but there is a larger space of configurations in which the updates improve rather than hinder the algorithm’s performance.
5.3 Mimic task: result and analysis
For the Mimic task (Fig. 5), we see similar clustering of the algorithms but not in the same order. RTRL, F-BPTT, and KF-RTRL perform the best. UORO and R-KF-RTRL perform similarly to each other, but relatively worse than they did on Add. Conversely, DNI, DNI(b), KeRNL, and RFLO perform relatively better on this task than they did on Add. The information content that must be memorized for Add is relatively small (), but the time horizon is quite long. In Mimic, it is difficult to quantify the exact time horizon, but it is clear that the network must memorize much more information (). Thus perhaps UORO and R-KF-RTRL are effective at maintaining information over time, but the stochasticity in the updates places a limit on how much information can be retained.
5.4 Gradient similarity analysis
We conduct an in-depth investigation looking beyond task accuracy by directly comparing the gradients (or approximate gradients) produced by each algorithm. Fig. 6 shows how a given pair of algorithms align on a time-step-by-time-step basis for the Add and Mimic tasks. Each subplot is a histogram giving the distribution of normalized dot products
for (flattened) weight updates and prescribed by algorithms and , respectively, at time .
Figs. 6 a,c show qualitatively similar trends:
As shown directly in Figs. 6b,d, PF algorithms (UORO, KF-RTRL, R-KF-RTRL, RFLO, KeRNL) align better with RTRL than with F-BPTT, and vice versa for FF algorithms (DNI).
The deterministic PF algorithms (RFLO and KeRNL) align better with RTRL than the stochastic algorithms (UORO, KF-RTRL, and R-KF-RTRL) align with RTRL.
RFLO and KeRNL align more strongly with each other than any other pair.
UORO and R-KF-RTRL do not align strongly with any other algorithms, despite their ability to train the RNN effectively. UORO’s mean alignments with RTRL are 0.043 on Add and 0.084 on Mimic, while R-KF-RTRL’s mean alignments with RTRL are 0.050 on Add and 0.092 on Mimic (Fig. 6b,d), which are much lower than all other approximate algorithms, even those that perform worse on the task in some cases.
Observations (i)-(iii) validate our categorizations, as similarity according to the normalized alignment corresponds to similarity by the past-facing/future-facing, tensor structure, and stochastic/deterministic criteria. Observation (iv) is puzzling, as it shows that angular alignment with exact algorithms is not predictive of learning performance.
How are UORO and R-KF-RTRL able to learn at all if their gradients are almost orthogonal with RTRL on average? We address this question for both UORO and R-KF-RTRL by examining the joint distribution of the gradient’s alignment with RTRL and the gradient’s norm (Fig. 7). All 4 cases show a statistically significant positive linear correlation between the normalized alignment and the common log of the gradient norm. This observation may partially explain (iv), because larger weight updates occur when UORO happens to strongly align with RTRL. However, these correlations are fairly weak even if statistically significant, and we argue that better algorithm similarity metrics are needed to account for observed differences in performance.
5.5 RFLO analysis
Among all approximate algorithms, RFLO stands out as having the simplest tensor structure and update rule, and it has been empirically demonstrated to be able to train RNNs on long-term dependencies. This is such a severe approximation of RTRL, yet it works so well in practice—and there is no clear understanding why. Although Murray (2019) goes into detail showing how loss decreases despite the random feedback used to approximately calculate , he does not address the more basic mystery of how RFLO is able to learn despite the significant approximation . In this section, we provide some intuition for how this simple learning rule is so successful and empirically validate our claims.
We hypothesize that, rather than learning dynamics that actively retain useful bits of the past like RTRL and BPTT, RFLO works by training what is essentially a high-capacity feedforward model to predict labels from the natural memory traces of previous inputs contained in the hidden state. This is reminiscient of reservoir computing (Lukoševičius and Jaeger, 2009). We illustrate this idea in the special case of a perfectly discrete network (), where the learning rule still performs remarkably well despite containing no network history. As Fig. 8a depicts, ultimately maps to via a single-hidden-layer feedforward network parameterized by and .
The RFLO learning rule in the discrete case corresponds exactly to training by backpropagation:
While every learning algorithm additionally trains online to best map to , this purely linear model cannot perfectly capture the complex ways that information about past inputs , implicit in relates to labels . Adding a hidden layer improves the ability of the network to predict from whatever evidence of is naturally retained in , analogous to how a single-hidden-layer feedforward network outperforms a simple softmax regression on MNIST (Deng, 2012).
To empirically validate our explanation, we first show that the strength of natural memory traces in the RNN depends on its recurrent weights. We measure this “memory” by running an untrained RNN with fixed weights forwards for k time steps of the Add task and calculating the value of a linear regression of onto for different values of the time shift . The sudden jumps in information occur as passes the time lags of the input-output dependencies explicitly included in the task (), followed by a slow decay as the information relevant for predicting gets corrupted by running the network forwards. The speed of this decay differs by the choice of . As Fig. 8b shows, orthogonal and Gaussian seem to best preserve information over time, while symmetric and diagonal lose information quite rapidly, likely due to their real spectra. In separate simulations (Fig. 8c), we trained networks initialized in these ways using the RFLO rule, and only the networks initialized with orthogonal or Gaussian are able to learn at all with RFLO. This validates our hypothesis, that RFLO works by a static prediction of based on evidence in , because in cases where this evidence is absent (or at least weak) due to rapid decay, RFLO fails.
We have presented a framework for conceptually unifying many recent advances in efficient online training of recurrent neural networks. These advances come from multiple perspectives: pure machine learning, biologically plausible machine learning, and computational neuroscience. We started by articulating a crucial distinction, (a) past facing vs. future facing, that divides algorithms by what exactly they calculate. We then presented a few other properties that characterize each new technique: (b) the tensor structure of the algorithm, (c) whether its update requires explicit randomness, and (d) whether its update can be derived in closed form from the relations (5)–(7) or must be approximated with SGD or some other numerical approach. Along the way, we clarified the relationship between some of the modern online approximations and exact methods. Further, we showed it’s possible to create new algorithms by identifying unexplored combinations of properties (a)-(d).
We empirically validated these ideas with synthetic tasks on which all algorithms could perform reasonably well in an online setting, despite using vanilla RNN architecture; gradient descent optimization with fixed learning rate; and no standard machine learning techniques such as gradient clipping, batch/layer normalization, regularization, etc. We saw that training errors roughly cluster according to their categorical distinctions, across tasks and hyperparameter choices. But performance of these clusters differed across tasks: for example, UORO and R-KF-RTRL performed relatively well (poorly) on Add (Mimic), while for DNI and DNI(b) this effect was flipped. We speculate that this can be explained in terms of the task demands: the Add task has a long time horizon (10 time steps) of explicit dependencies for small inputs (), while the Mimic task requires more information to be memorized () that decays with the target RNN’s forward pass (harder to measure, but likely shorter than 10 time steps). Thus perhaps UORO and R-KF-RTRL produce gradients with a limited amount of information that survives many updates, while DNI and DNI(b) have a larger information capacity but a limited time horizon. Future work should more systematically explore how these different task features interact with the different algorithmic approximations.
Following common practice, we used the pairwise vector alignments of the (approximate) gradients calculated by each algorithm as a way to analyze the precision of different approximations. This similarity turned out to reflect the natural clustering of the algorithms along the axes proposed here. In particular, past-facing approximations had stronger alignment with the exact past-facing gradients calculated by RTRL compared to the exact (up to truncation) future-facing gradients calculated by F-BPTT, and vice versa for the future-facing approximation, DNI. Reassuringly, KeRNL and RFLO, which have the same tensor structure, featured strong alignment.
Importantly, the angular alignment of the gradients did not account for task performance: UORO and R-KF-RTRL performed quite well despite their weak alignment with RTRL and BPTT, while KeRNL performed relatively poorly despite its strong alignment with RTRL and BPTT. Analyzing the magnitudes of the gradients partially explained this observation, as UORO and R-KF-RTRL aligned with RTRL more strongly when their gradients were larger in norm. However, this effect was subtle, so probably not enough to account for the performance differences. This exposes a limitation of current ways of analyzing this class of algorithms. There clearly is a need for better similarity metrics that go beyond time-point-wise gradient alignment and instead compare long-term learning trajectories.
Notably, it is the stochastic algorithms that have particularly weak alignment with RTRL—even KF-RTRL, yet its performance is indistinguishable from the exact algorithms. Averaged over many time steps of learning, corresponding to many samples of , these stochastic methods do contain complete information about (i.e. are unbiased), but at any one time point the alignment is heavily corrupted by the explicit randomness. In contrast, deterministic approximations, such as KeRNL, RFLO and DNI, may partially align with exact methods by construction, but their errors have no reason to average out, hence their inability to find the same minima as exact methods (Fig. 9). This may also explain why stochastic approximations do not align with each other despite their conceptual similarity.
We chose the simplest possible setup for comparison, using basic RNN architecture and gradient descent, aiming to be as inclusive as possible of various algorithms. Of course, there are alternative online methods that do not naturally fall into this framework. Any online algorithm that exploits a particular RNN architecture necessarily cannot fit into our classification. For example, Ororbia et al. (2017, 2018) propose a specialized neural architecture (Temporal Neural Coding Network), whose learning algorithm (Discrepancy Reduction) is naturally local, due to the network structure. In contrast, the algorithms reviewed here were explicitly derived as approximations to nonlocal algorithms (RTRL and BPTT), and their locality ends up being more of a bug than a feature, as evidenced by their impaired performance. From a machine learning perspective, there are limits to general-purpose approximations of this kind, and future progress in online methods will likely come about by further exploring specialized algorithm-architecture pairs.
The same is true from a biological perspective. In the brain, cortical architecture and synaptic plasticity rules evolved together, while physical constraints dictate that plasticity is necessarily local. Still, the details of how local plasticity rules interact with neural circuits remain mysterious, and this is a current focus of research (Guerguiev et al., 2017; Sacramento et al., 2018; Lillicrap and Santoro, 2019). Exploring which architectures allow locality to manifest as a consequence, rather than a constraint, of learning is a potentially fruitful point of interaction between artificial intelligence and computational neuroscience.
We thank Guangyu Yang and James Murray for helpful conversations.
- Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
- Bahdanau et al. (2016) Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End-to-end attention-based large vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4945–4949. IEEE, 2016.
- Benzing et al. (2019) Frederik Benzing, Marcelo Matheus Gauy, Asier Mujika, Anders Martinsson, and Angelika Steger. Optimal kronecker-sum approximation of real time recurrent learning. arXiv preprint arXiv:1902.03993, 2019.
- Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
- Cho et al. (2015) Kyunghyun Cho, Aaron Courville, and Yoshua Bengio. Describing multimedia content using attention-based encoder-decoder networks. IEEE Transactions on Multimedia, 17(11):1875–1886, 2015.
- Cooijmans and Martens (2019) Tim Cooijmans and James Martens. On the variance of unbiased online recurrent optimization. arXiv preprint arXiv:1902.02405, 2019.
- Czarnecki et al. (2017) Wojciech Marian Czarnecki, Grzegorz Swirszcz, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 904–912. JMLR. org, 2017.
- Deng (2012) Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
- Graves (2013) Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.
- Graves et al. (2016) Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471, 2016.
- Guerguiev et al. (2017) Jordan Guerguiev, Timothy P Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites. ELife, 6:e22901, 2017.
- Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
- Jaderberg et al. (2017) Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1627–1635. JMLR. org, 2017.
- Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Lillicrap and Santoro (2019) Timothy P Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current Opinion in Neurobiology, 55:82 – 89, 2019. ISSN 0959-4388. doi: https://doi.org/10.1016/j.conb.2019.01.011. URL http://www.sciencedirect.com/science/article/pii/S0959438818302009. Machine Learning, Big Data, and Neuroscience.
- Lillicrap et al. (2016) Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7:13276, 2016.
- Lukoševičius and Jaeger (2009) Mantas Lukoševičius and Herbert Jaeger. Reservoir computing approaches to recurrent neural network training. Computer Science Review, 3(3):127–149, 2009.
- Marschall et al. (2019) Owen Marschall, Kyunghyun Cho, and Cristina Savin. Using local plasticity rules to train recurrent neural networks. In Computational and Systems Neuroscience (CoSyNe), number II-90, Lisbon, Portugal, Feb 2019.
- Mikolov et al. (2010) Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
- Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
- Mujika et al. (2018) Asier Mujika, Florian Meier, and Angelika Steger. Approximating real-time recurrent learning with random kronecker factors. In Advances in Neural Information Processing Systems, pages 6594–6603, 2018.
- Murray (2019) James M Murray. Local online learning in recurrent networks with random feedback. eLife, 8:e43299, 2019.
- Ororbia et al. (2018) Alexander Ororbia, Ankur Mali, C Lee Giles, and Daniel Kifer. Online learning of recurrent neural architectures by locally aligning distributed representations. arXiv preprint arXiv:1810.07411, 2018.
- Ororbia et al. (2017) II Ororbia, G Alexander, Patrick Haffner, David Reitter, and C Lee Giles. Learning to adapt by minimizing discrepancy. arXiv preprint arXiv:1711.11542, 2017.
- Pitis (2016) Silviu Pitis. Recurrent neural networks in tensorflow 1. r2rt.com/recurrent-neural-networks-in-tensorflow-i, 2016. Accessed: 2018-11-13.
- Roth et al. (2019) Christopher Roth, Ingmar Kanitscheider, and Ila Fiete. Kernel RNN learning (keRNL). In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=ryGfnoC5KQ.
- Sacramento et al. (2018) João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems, pages 8721–8732, 2018.
- Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
- Tallec and Ollivier (2017) Corentin Tallec and Yann Ollivier. Unbiased online recurrent optimization. arXiv preprint arXiv:1702.05043, 2017.
- Tieleman and Hinton (2012) T. Tieleman and G. Hinton. Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
- Werbos et al. (1990) Paul J Werbos et al. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560, 1990.
- Williams and Zipser (1989) Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270–280, 1989.
Lemma for generating rank-1 unbiased estimates
For completeness, we state the Lemma from Tallec and Ollivier (2017) in components notation. Given a decomposition of a matrix into rank-1 components
a vector of i.i.d. random variables with , , and a list of positive constants , then
is a rank-1, unbiased estimate of over the choice of .
For reproducibility, we describe in fuller detail our implementation configurations for each simulation. Table 2 shows hyperparameter/configuration choices that apply across all algorithms. Table 3 shows the algorithm-specific hyperparameter choices we made for each task. In Table 2, we reference sub-matrices of and , since they are initialized differently.
|learning rate||learning rate for SGD w.r.t. and|
|32||number of hidden units in the network|
|tanh||nonlinearity used in RNN forward dynamics|
|init.||initial value for input weights|
|init.||rand. orth.||initial value for recurrent weights|
|init.||0||initial value for recurrent bias|
|init.||initial value for output weights|
|init.||0||initial value for output bias|
|init.||value for fixed feedback weights used in DNI(b)|
|init.||initial value for target recurrent bias in Mimic|
|init.||initial value for target output bias in Mimic|
Some miscellaneous implementation details below:
For the Add task in the condition, we changed the DNI/DNI(b) learning rate to for and for (DNI(b)). In other cases, the learning rates for and are identical.
There are two appearances of the synthetic gradient weights in Eq. (26). Although we wrote them as one matrix for brevity, in implementation we actually keep two separate values, and , the latter of which we use for for the right-hand appearance (specifically to calculate the bootstrapped estimate of the SG training label). We update every time step but keep constant, replacing it with the latest value of only once per time steps. This integer introduces another hyperparameter, which we choose to be 5. (Inspired by an analogous technique used in deep Q-learning from Mnih et al., 2015.)
In the original paper, Roth et al. (2019) use rather than as a temporal filter for . We made this change so that makes sense in terms of the in the forward dynamics of the network and RFLO. Of course, these are equivalent via , but the gradient w.r.t. must be rescaled by a factor of to compensate.
For KeRNL, there is a choice for how to update the eligibility trace (Eq. 17): one can scale the right-hand term by either the learned timescale or the RNN timescale . We chose the latter because it has stronger empirical performance and it theoretically recovers the RTRL equation under the approximating assumptions about .
Perturbations for calculating gradients for and in KeRNL are sampled i.i.d. .
In our implementation of the Add task, we use for a “one-hot” representation of the input and label , such that and .
In our implementation of Mimic, the target RNN was initialized in the same way as the RNNs we train, with the exception of the recurrent and output biases (see Table 2).