Abstract
Entropy regularization is an important idea in reinforcement learning, withgreat success in recent algorithms like Soft Q Network (SQN) and SoftActor-Critic (SAC1). In this work, we extend this idea into the on-policyrealm. We propose the soft policy gradient theorem (SPGT) for on-policy maximumentropy reinforcement learning. With SPGT, a series of new policy optimizationalgorithms are derived, such as SPG, SA2C, SA3C, SDDPG, STRPO, SPPO, SIMPALAand so on. We find that SDDPG is equivalent to SAC1. For policy gradient, thepolicy network is often represented as a Gaussian distribution with a globalaction variance, which damages the representation capacity. We introduce alocal action variance for policy network and find it can work collaborativelywith the idea of entropy regularization. Our method outperforms prior works ona range of benchmark tasks. Furthermore, our method can be easily extended tolarge scale experiment with great stability and parallelism.