Abstract
Lifelong learning is the problem of learning multiple consecutive tasks in asequential manner, where knowledge gained from previous tasks is retained andused to aid future learning over the lifetime of the learner. It is essentialtowards the development of intelligent machines that can adapt to theirsurroundings. In this work we focus on a lifelong learning approach tounsupervised generative modeling, where we continuously incorporate newlyobserved distributions into a learned model. We do so through a student-teacherVariational Autoencoder architecture which allows us to learn and preserve allthe distributions seen so far, without the need to retain the past data nor thepast models. Through the introduction of a novel cross-model regularizer,inspired by a Bayesian update rule, the student model leverages the informationlearned by the teacher, which acts as a probabilistic knowledge store. Theregularizer reduces the effect of catastrophic interference that appears whenwe learn over sequences of distributions. We validate our model's performanceon sequential variants of MNIST, FashionMNIST, PermutedMNIST, SVHN and Celeb-Aand demonstrate that our model mitigates the effects of catastrophicinterference faced by neural networks in sequential learning scenarios.