Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods

  • 2021-02-25 17:55:15
  • Nicholay Topin, Stephanie Milani, Fei Fang, Manuela Veloso
  • 3

Abstract

Current work in explainable reinforcement learning generally producespolicies in the form of a decision tree over the state space. Such policies canbe used for formal safety verification, agent behavior prediction, and manualinspection of important features. However, existing approaches fit a decisiontree after training or use a custom learning procedure which is not compatiblewith new learning techniques, such as those which use neural networks. Toaddress this limitation, we propose a novel Markov Decision Process (MDP) typefor learning decision tree policies: Iterative Bounding MDPs (IBMDPs). An IBMDPis constructed around a base MDP so each IBMDP policy is guaranteed tocorrespond to a decision tree policy for the base MDP when using amethod-agnostic masking procedure. Because of this decision tree equivalence,any function approximator can be used during training, including a neuralnetwork, while yielding a decision tree policy for the base MDP. We present therequired masking procedure as well as a modified value update step which allowsIBMDPs to be solved using existing algorithms. We apply this procedure toproduce IBMDP variants of recent reinforcement learning methods. We empiricallyshow the benefits of our approach by solving IBMDPs to produce decision treepolicies for the base MDPs.

 

Quick Read (beta)

loading the full paper ...