Abstract
This paper investigates how to efficiently transition and update policies,trained initially with demonstrations, using off-policy actor-criticreinforcement learning. It is well-known that techniques based on Learning fromDemonstrations, for example behavior cloning, can lead to proficient policiesgiven limited data. However, it is currently unclear how to efficiently updatethat policy using reinforcement learning as these approaches are inherentlyoptimizing different objective functions. Previous works have used lossfunctions which combine behavioral cloning losses with reinforcement learninglosses to enable this update, however, the components of these loss functionsare often set anecdotally, and their individual contributions are not wellunderstood. In this work we propose the Cycle-of-Learning (CoL) framework thatuses an actor-critic architecture with a loss function that combines behaviorcloning and 1-step Q-learning losses with an off-policy pre-training step fromhuman demonstrations. This enables transition from behavior cloning toreinforcement learning without performance degradation and improvesreinforcement learning in terms of overall performance and training time.Additionally, we carefully study the composition of these combined losses andtheir impact on overall policy learning. We show that our approach outperformsstate-of-the-art techniques for combining behavior cloning and reinforcementlearning for both dense and sparse reward scenarios. Our results also suggestthat directly including the behavior cloning loss on demonstration data helpsto ensure stable learning and ground future policy updates.