Abstract
Catastrophic forgetting has remained a significant challenge for efficientreinforcement learning for decades (Ring 1994, Rivest and Precup 2003). Whilerecent works have proposed effective methods to mitigate this issue, theymainly focus on the algorithmic side. Meanwhile, we do not fully understandwhat architectural properties of neural networks lead to catastrophicforgetting. This study aims to fill this gap by studying the role of activationfunctions in the training dynamics of neural networks and their impact oncatastrophic forgetting in reinforcement learning setup. Our study revealsthat, besides sparse representations, the gradient sparsity of activationfunctions also plays an important role in reducing forgetting. Based on thisinsight, we propose a new class of activation functions, elephant activationfunctions, that can generate both sparse outputs and sparse gradients. We showthat by simply replacing classical activation functions with elephantactivation functions in the neural networks of value-based algorithms, we cansignificantly improve the resilience of neural networks to catastrophicforgetting, thus making reinforcement learning more sample-efficient andmemory-efficient.