LASER: Learning a Latent Action Space for Efficient Reinforcement Learning

  • 2021-03-30 12:19:29
  • Arthur Allshire, Roberto Martín-Martín, Charles Lin, Shawn Manuel, Silvio Savarese, Animesh Garg
  • 0

Abstract

The process of learning a manipulation task depends strongly on the actionspace used for exploration: posed in the incorrect action space, solving a taskwith reinforcement learning can be drastically inefficient. Additionally,similar tasks or instances of the same task family impose latent manifoldconstraints on the most effective action space: the task family can be bestsolved with actions in a manifold of the entire action space of the robot.Combining these insights we present LASER, a method to learn latent actionspaces for efficient reinforcement learning. LASER factorizes the learningproblem into two sub-problems, namely action space learning and policy learningin the new action space. It leverages data from similar manipulation taskinstances, either from an offline expert or online during policy learning, andlearns from these trajectories a mapping from the original to a latent actionspace. LASER is trained as a variational encoder-decoder model to map rawactions into a disentangled latent action space while maintaining actionreconstruction and latent space dynamic consistency. We evaluate LASER on twocontact-rich robotic tasks in simulation, and analyze the benefit of policylearning in the generated latent action space. We show improved sampleefficiency compared to the original action space from better alignment of theaction space to the task space, as we observe with visualizations of thelearned action space manifold. Additional details:https://www.pair.toronto.edu/laser

 

Quick Read (beta)

loading the full paper ...