An Efficient, Generalized Bellman Update For Cooperative Inverse Reinforcement Learning

  • 2018-06-11 06:06:43
  • Dhruv Malik, Malayandi Palaniappan, Jaime F. Fisac, Dylan Hadfield-Menell, Stuart Russell, Anca D. Dragan
  • 0

Abstract

Our goal is for AI systems to correctly identify and act according to theirhuman user's objectives. Cooperative Inverse Reinforcement Learning (CIRL)formalizes this value alignment problem as a two-player game between a humanand robot, in which only the human knows the parameters of the reward function:the robot needs to learn them as the interaction unfolds. Previous work showedthat CIRL can be solved as a POMDP, but with an action space size exponentialin the size of the reward parameter space. In this work, we exploit a specificproperty of CIRL---the human is a full information agent---to derive anoptimality-preserving modification to the standard Bellman update; this reducesthe complexity of the problem by an exponential factor and allows us to relaxCIRL's assumption of human rationality. We apply this update to a variety ofPOMDP solvers and find that it enables us to scale CIRL to non-trivialproblems, with larger reward parameter spaces, and larger action spaces forboth robot and human. In solutions to these larger problems, the human exhibitspedagogic (teaching) behavior, while the robot interprets it as such andattains higher value for the human.

 

Quick Read (beta)

loading the full paper ...