Multi-Task Reinforcement Learning with Soft Modularization

  • 2020-03-30 17:47:04
  • Ruihan Yang, Huazhe Xu, Yi Wu, Xiaolong Wang
  • 3

Abstract

Multi-task learning is a very challenging problem in reinforcement learning.While training multiple tasks jointly allow the policies to share parametersacross different tasks, the optimization problem becomes non-trivial: It isunclear what parameters in the network should be reused across tasks, and thegradients from different tasks may interfere with each other. Thus, instead ofnaively sharing parameters across tasks, we introduce an explicitmodularization technique on policy representation to alleviate thisoptimization issue. Given a base policy network, we design a routing networkwhich estimates different routing strategies to reconfigure the base networkfor each task. Instead of creating a concrete route for each task, ourtask-specific policy is represented by a soft combination of all possibleroutes. We name this approach soft modularization. We experiment with multiplerobotics manipulation tasks in simulation and show our method improves sampleefficiency and performance over baselines by a large margin.

 

Quick Read (beta)

loading the full paper ...