We consider the problem of building a state representation model in acontinual fashion. As the environment changes, the aim is to efficientlycompress the sensory state's information without losing past knowledge. Thelearned features are then fed to a Reinforcement Learning algorithm to learn apolicy. We propose to use Variational Auto-Encoders for state representation,and Generative Replay, i.e. the use of generated samples, to maintain pastknowledge. We also provide a general and statistically sound method forautomatic environment change detection. Our method provides efficient staterepresentation as well as forward transfer, and avoids catastrophic forgetting.The resulting model is capable of incrementally learning information withoutusing past data and with a bounded system size.