Invariant Transform Experience Replay: Data Augmentation for Deep Reinforcement Learning

  • 2020-07-05 03:19:06
  • Yijiong Lin, Jiancong Huang, Matthieu Zimmer, Yisheng Guan, Juan Rojas, Paul Weng
  • 0

Abstract

Deep Reinforcement Learning (RL) is a promising approach for adaptive robotcontrol, but its current application to robotics is currently hindered by highsample requirements. To alleviate this issue, we propose to exploit thesymmetries present in robotic tasks. Intuitively, symmetries from observedtrajectories define transformations that leave the space of feasible RLtrajectories invariant and can be used to generate new feasible trajectories,which could be used for training. Based on this data augmentation idea, weformulate a general framework, called Invariant Transform Experience Replaythat we present with two techniques: (i) Kaleidoscope Experience Replayexploits reflectional symmetries and (ii) Goal-augmented Experience Replaywhich takes advantage of lax goal definitions. In the Fetch tasks from OpenAIGym, our experimental results show significant increases in learning rates andsuccess rates. Particularly, we attain a 13, 3, and 5 times speedup in thepushing, sliding, and pick-and-place tasks respectively in the multi-goalsetting. Performance gains are also observed in similar tasks with obstaclesand we successfully deployed a trained policy on a real Baxter robot. Our workdemonstrates that invariant transformations on RL trajectories are a promisingmethodology to speed up learning in deep RL.

 

Quick Read (beta)

loading the full paper ...