Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation

  • 2019-06-07 16:16:41
  • Ruohan Wang, Carlo Ciliberto, Pierluigi Amadori, Yiannis Demiris
  • 0

Abstract

We consider the problem of imitation learning from a finite set of experttrajectories, without access to reinforcement signals. The classical approachof extracting the expert's reward function via inverse reinforcement learning,followed by reinforcement learning is indirect and may be computationallyexpensive. Recent generative adversarial methods based on matching the policydistribution between the expert and the agent could be unstable duringtraining. We propose a new framework for imitation learning by estimating thesupport of the expert policy to compute a fixed reward function, which allowsus to re-frame imitation learning within the standard reinforcement learningsetting. We demonstrate the efficacy of our reward function on both discreteand continuous domains, achieving comparable or better performance than thestate of the art under different reinforcement learning algorithms.

 

Quick Read (beta)

Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation

Ruohan Wang1 Carlo Ciliberto1  Pierluigi Amadori1  Yiannis Demiris1
{r.wang16,c.ciliberto,p.amadori,y.demiris}@imperial.ac.uk
Abstract

We consider the problem of imitation learning from a finite set of expert trajectories, without access to reinforcement signals. The classical approach of extracting the expert’s reward function via inverse reinforcement learning, followed by reinforcement learning is indirect and may be computationally expensive. Recent generative adversarial methods based on matching the policy distribution between the expert and the agent could be unstable during training. We propose a new framework for imitation learning by estimating the support of the expert policy to compute a fixed reward function, which allows us to re-frame imitation learning within the standard reinforcement learning setting. We demonstrate the efficacy of our reward function on both discrete and continuous domains, achieving comparable or better performance than the state of the art under different reinforcement learning algorithms.

1 Introduction

11footnotetext: Department of Electrical and Electronic Engineering, Imperial College London, SW7 2BT, United Kingdom

We consider a specific setting of imitation learning - the task of policy learning from expert demonstrations - in which the learner only has a finite number of expert trajectories without any further access to the expert. Two broad categories of approaches to this settings are behavioral cloning (BC) Pomerleau (1991), which directly learns a policy mapping from states to actions with supervised learning from expert trajectories; and inverse reinforcement learning (IRL) Ng & Russell (2000); Abbeel & Ng (2004), which learns a policy via reinforcement learning, using a cost function extracted from expert trajectories.

Most notably, BC has been successfully applied to the task of autonomous driving Bojarski et al. (2016); Bansal et al. (2018). Despite its simplicity, BC typically requires a large amount of training data to learn good policies, as it may suffer from compounding errors caused by covariate shift Ross & Bagnell (2010); Ross et al. (2011). BC is often used as a policy initialization step for further reinforcement learning Nagabandi et al. (2018); Rajeswaran et al. (2017).

IRL estimates a cost function from expert trajectories and uses reinforcement learning to derive policies. As the cost function evaluates the quality of trajectories rather than that of individual actions, IRL avoids the problem of compounding errors. IRL is effective with a wide range of problems, from continuous control benchmarks in the Mujoco environment Ho & Ermon (2016), to robot footsteps planning Ziebart et al. (2008).

Generative Adversarial Imitation Learning (GAIL) Ho & Ermon (2016); Baram et al. (2017) connects IRL to the general framework of Generative Adversarial Networks (GANs) Goodfellow et al. (2014), and re-frames imitation learning as distribution matching between the expert policy and the learned policy via Jensen-Shannon Divergence. However, GAIL naturally inherits the challenges of GAN training, including possible training instability, and overfitting to training data Arjovsky & Bottou (2017); Brock et al. (2018). Similarly, generative moment matching imitation learning (GMMIL) considers distribution matching via maximum mean discrepancy. Both GAIL and GMMIL iteratively update the reward function and the learned policy during training.

In this paper, we propose imitation learning via expert policy support estimation, a new general framework for recovering a reward function from expert trajectories. We propose Random Expert Distillation (RED) by providing a connection between Random Network Distillation (RND) Burda et al. (2018) – a method to design intrinsic rewards for RL exploration based on the "novelty" of states visited – and support estimation ideas. Our method computes a fixed reward function from expert trajectories, and obviates the need for dynamic update of reward functions found in classical IRL, GAIL and GMMIL. Evaluating RED using different reinforcement learning algorithms on both discrete and continuous domains, we show that the proposed method achieves comparable or better performance than the state-of-arts methods on a variety of tasks, including a driving scenario (see Figure 3(b)). To the best of our knowledge, our method is the first to explore expert policy support estimation for imitation learning.

2 Background

We review the formulation of Markov Decision Process (MDP) in the context of which we consider imitation learning. We also review previous approaches to imitation learning and support estimation related to our proposed method.


Setting. We consider an infinite-horizon discounted MDP, defined by the tuple (S,A,P,r,p0,γ), where S is the set of states, A the set of actions, P:S×A×S[0,1] the transition probability, r:S×A the reward function, p0:S[0,1] the distribution over initial states, and γ(0,1) the discount factor. Let π be a stochastic policy π:S×A[0,1] with expected discounted reward 𝔼π(r(s,a))𝔼(t=0γtr(st,at)) where s0p0, atπ(|st), and st+1P(|st,at) for t0. We denote πE the expert policy.

2.1 Imitation Learning

We briefly review the main methods for imitation learning:


Behavioral Cloning (BC) learns a control policy π:SA directly from expert trajectories via supervised learning. Despite its simplicity, BC is prone to compounding errors: small mistakes in the policy cause the agent to deviate from the state distribution seen during training, making future mistakes more likely. Mistakes accumulate, leading to eventual catastrophic errors Ross et al. (2011). While several strategies have been proposed to address this Ross & Bagnell (2010); Sun et al. (2017), they often require access to the expert policy during the entire training process, rather than a finite set of expert trajectories. BC is commonly used to initialize control policies for reinforcement learning Rajeswaran et al. (2017); Nagabandi et al. (2018).


Inverse Reinforcement Learning (IRL) models the expert trajectories with a Boltzmann distribution Ziebart et al. (2008), where the likelihood of a trajectory is defined as:

pθ(τ)=1Zexp(-cθ(τ)) (1)

where τ={s1,a1,s2,a2} is a trajectory, cθ(τ)=tcθ(st,at) a learned cost function parametrized by θ, and the partition function Zexp(-cθ(τ))d(τ) the integral over all trajectories consistent with transition function of the MDP. The main computational challenge of IRL is the efficient estimation the partition function Z. IRL algorithms typically optimize the cost function by alternating between updating the cost function and learning an optimal policy with respect to the current cost function with reinforcement learning Abbeel & Ng (2004); Ng & Russell (2000).


Imitation Learning via Distribution Matching frames imitation learning as distribution matching between the distribution of state-action pairs of the expert policy and that of the learned policy. In Ho & Ermon (2016); Finn et al. (2016), the authors connect IRL to distribution matching via Generative Adversarial Networks (GANs) Goodfellow et al. (2014). Known as Generative Adversarial Imitation Learning (GAIL), the method imitates an expert policy by formulating the following minimax game:

minπmaxD(0,1)𝔼π(logD(s,a))+𝔼πE(log(1-D(s,a)))-λH(π) (2)

where the expectations 𝔼π and 𝔼πE denote the joint distributions over state-action pairs of the learned policy and the expert policy, respectively, and H(π)𝔼π(-logπ(a|s)) is the entropy of the learned policy. GAIL has been successfully applied to various control tasks in the Mujoco environment Ho & Ermon (2016); Baram et al. (2017). However, GAIL inherits the challenges of GANs, including possible training instability such as vanishing or exploding gradients, as well as overfitting to training data Arjovsky & Bottou (2017); Brock et al. (2018). While numerous theoretical and practical techniques have been proposed to improve GANs (e.g Arjovsky et al. (2017); Salimans et al. (2016)), a large-scale study of GANs show that many GAN algorithms and architectures achieve similar performance with sufficient hyperparameter tuning and random restarts, and no algorithm or network architecture stands out as the clear winner on all tasks Lucic et al. (2018).

Similar to GAIL, Kim & Park (2018) proposed generative moment matching imitation learning (GMMIL) by minimizing the maximum mean discrepancy between the expert policy and the learned policy. Though GMMIL avoids the difficult minimax game, the cost of each reward function evaluation grows linearly with the amount of training data, which makes scaling the method to large dataset potentially difficult. In addition, we demonstrate in our experiments that GMMIL may fail to estimate the appropriate reward functions.

2.2 Support Estimation with Kernel Methods

As we will motivate in detail in Sec. 3, we argue that estimating the support of the expert policy can lead to good reward functions for imitation learning. In this section we review one of the most well-established approaches to support estimation of a distribution from a finite number of i.i.d. samples, which relies on a kernelized version of principal component analysis Schölkopf et al. (1998). The idea is to leverage the Separating Property De Vito et al. (2014) of suitable reproducing kernel Hilbert spaces, which guarantees the covariance operator of the embedded data to precisely capture the geometrical properties of the support of the underlying distribution. This allows us to derive a test function that is zero exclusively on points belonging to the support of the distribution.

Formally, let 𝒳d be a set (in our setting 𝒳=S×A is the joint state-action space) and let k:𝒳×𝒳 be a positive definite kernel with associated reproducing kernel Hilbert space (RKHS) Aronszajn (1950) and feature map ϕ:𝒳, such that k(x,x)=ϕ(x),ϕ(x) for any x,x𝒳. For any set U𝒳, denote by Φ(U)={ϕ(x)|xU} and Φ(U)¯ the closure of its span in . The separating property guarantees that for any closed subset U of 𝒳, Φ(U)=Φ(𝒳)Φ(U)¯. In other words, the separating property ensures that

xUϕ(x)Φ(U)¯. (3)

As shown in De Vito et al. (2014), several kernels allow the separating property to hold, including the popular Gaussian kernel k(x,x)=exp(-x-x/σ) for any σ>0.

The separating property suggests a natural strategy to test whether a point x𝒳 belongs to a closed set U. Let PU: be the orthogonal projection operator onto Φ(U)¯ (which is a linear operator since Φ(U)¯ is a linear space), we have

xU(I-PU)ϕ(x)=0,

since Eq. (3) corresponds to ϕ(x)=PUϕ(x), or equivalently (I-PU)ϕ(x)=0.

We can leverage the machinery introduced above to estimate the support supp(π)𝒳 of a probability distribution π on 𝒳. We observe that the covariance operator Cπ=𝔼π[ϕ(x)ϕ(x)] encodes sufficient information to recover the orthogonal projector Psupp(π) (denoted Pπ in the following for simplicity). More precisely, let Cπ denote the pseudoinverse of Cπ. A direct consequence of the separating property is that Pπ=CπCπ De Vito et al. (2014).

When π is unknown and observed only through a finite number of N i.i.d examples {xi}i=1N, it is impossible to obtain Cπ (and thus Pπ) exactly. A natural strategy is to consider the empirical covariance operator C^=1Ni=1Nϕ(xi)ϕ(xi) as a proxy of the ideal one. Then, the projector is estimated as P^=C^mC^m, where Cm is the operator comprising the mn principal component directions of C, where m is a hyperparameter to control the stability of the pseudoinverse when computing P^. We can then test whether a point x𝒳 belongs to supp(π), by determining if

(I-P^)ϕ(x)2=ϕ(x),(I-P^)ϕ(x)=k(x,x)-ϕ(x),P^ϕ(x), (4)

is greater than zero (in practice a threshold τ>0 is introduced). Note that we have used the fact that P^P^=P^, since P^ is an orthogonal projector. Although C^ and P^ are operators between possibly infinite dimensional spaces, we have for any x𝒳 (see Schölkopf et al., 1998)

ϕ(x),P^ϕ(x)=KxKmKx,

where Kn×n is the empirical kernel matrix of the training examples, with entries Kij=k(xi,xj). Here, Km denotes the matrix obtained by performing PCA on K and taking the first mn principal directions and Kxn is the vector with entries (Kx)i=k(xi,x). Therefore ϕ(x),P^ϕ(x) can be computed efficiently in practice.

The theoretical properties of the strategy described above have been thoroughly investigated in the previous literature De Vito et al. (2014); Rudi et al. (2017). In particular, it has been shown that the Hausdorff distance between supp(π) and the set induced by P^, tends to zero when n+, provided that the number of principal components m grows with the number of training examples. Moreover, it has been shown that the support estimator enjoys fast learning rate under standard regularity assumptions.

3 Imitation Learning via Expert Policy Support Estimation

Formally, we are interested in extracting a reward function r^(s,a) from a finite set of trajectories D={τi}i=1N produced by an expert policy πE within a MDP environment. Here each τi is a trajectory of state-action pairs of the form τi={s1,a1,s2,a2,,sT,aT}. Assuming that the expert trajectories are consistent with some unknown reward function r*(s,a), our goal is for a policy learned by applying RL to r^(s,a), to achieve good performance when evaluated against r*(s,a) (the standard evaluation strategy for imitation learning). In contrast with traditional imitation learning, we do not explicitly aim to match πE exactly.

We motivate our approach with a thought experiment. Given a discrete action space and a deterministic expert such that as*=πE(s), the resulting policy is a Dirac’s delta at each state sS, for all (s,a). Consider the reward function

rE(s,a)={1 if (s,a)supp(πE)0 if (s,a)supp(πE) (5)

which corresponds to the indicator function of supp(πE). It follows that a RL agent trained with this reward function would be able to imitate the expert exactly, since the discrete action space allows the random exploration of the agent to discover optimal actions at each state. In practice, the expert policy is unknown and only a finite number of trajectories sampled according to πE are available. In these context we can use support estimation techniques to construct a reward function r^.

For continuous action domains, sparse reward such as rE is unlikely to work since it is improbable for the agent, via random exploration, to discover the optimal actions, while all other actions are considered equally bad. Instead, since support estimation produces a score with Eq. (4) for testing each input, we may directly use that score to construct the reward function r^. The rationale is that actions similar to that of the expert in any given state would still receive high scores from support estimation, allowing the RL agent to discover those actions during exploration.

Based on the motivation above, we hypothesize that support estimation of the expert policy’s state-action distribution provides a viable reward function for imitation learning. Intuitively, the reward function encourages the RL agent to behave similarly as the expert at each state and remain on the estimated support of the expert policy. Further, this allows us to compute a fixed reward function based on the expert trajectories and re-frame imitation learning in the standard context of reinforcement learning.

We note that for stochastic experts, the support of πE might coincide with the whole space S×A. Support estimation would hence produce an uninformative, flat reward function rE when given an infinite amount of training data. However, we argue that an infinite amount of training data should allow BC to successfully imitate the expert, bypassing the intermediate step of extracting a reward function from the expert trajectories.

3.1 Practical Support Estimation

Following the strategy introduced in Sec. 2.2, we consider a novel approach to support estimation. Our method takes inspiration from kernel methods but applies to other models such as neural networks.

Let be a RKHS and f a function parametrized by θΘ. Consider the regression problem that admits a minimizer on

inff(fθ(x)-f(x))2𝑑π(x),

The minimal solution corresponds to fπ,θ=CπCπfθ=Pπfθ, the orthogonal projection of fθ onto the range of Cπ (see e.g Engl et al., 1996). For any x𝒳 we have

fθ(x)-fπ,θ(x)=fθ,(I-Pπ)ϕ(x).

In particular, if xsupp(π), we have (I-Pπ)ϕ(x)=0 and hence fθ(x)-fπ,θ(x)=0. The converse is unfortunately not necessarily true: for a point xsupp(π), (I-Pπ)ϕ(x) may still be orthogonal to fθ and thus fθ(x)-fπ,θ(x)=0.

A strategy to circumvent this issue is to consider multiple fθ, and then average their squared errors (fθ-fπ,θ)2. The rationale is that if xSπ, by spanning different directions in , at least one of the fθ will not be orthogonal to (I-Pπ)ϕ(x), which leads to a non-zero value when compared against fπ,θ(x). In particular, if we sample θ according to a probability distribution over Θ, we have

𝔼θ(fθ-fπ,θ)2 =𝔼θ(I-Pπ)ϕ(x),fθfθ(I-Pπ)ϕ(x)
=(I-Pπ)ϕ(x),Σ(I-Pπ)ϕ(x),

where Σ=𝔼θ[fθfθ] is the covariance of the random functions fθ generated from sampling the parameters θ. Ideally, we would like to sample θ such that Σ=I, which allows us to recover the support estimator (I-Pπ)ϕ(x) introduced in Sec. 2.2. However, it is not always clear how to sample fθ so that its covariance corresponds to the identity in practice. In this sense, this approach does not offer the same theoretical guarantees as the kernelized method reviewed in Sec. 2.2.

The discussion above provides us with a general strategy for support estimation. Consider a hypotheses space of functions fθ parametrized by θΘ (e.g. neural networks). We can sample θ1,,θK functions according to a distribution over Θ. Given a training dataset {xi}i=1N sampled from π, we solve the K regression problems

θ^k=minθΘ1Ni=1N(fθ(xi)-fθk(xi))2, (6)

for every k=1,,K. Then for any x𝒳, we can test whether it belongs to the support of π by checking whether

1Kk=1K(fθ^k(x)-fθk(x))2, (7)

is larger than a prescribed threshold. As K and N grow larger, the estimate above will approximate increasingly well the desired quantity 𝔼θ(fπ,θ(x)-fθ(x))2.

3.2 Reward Function with Random Network Distillation

We may interpret the recent method of Random Network Distillation (RND) Burda et al. (2018) as approximate support estimation. Formally, RND sets up a randomly generated prediction problem involving two neural networks: a fixed and randomly initialized target network fθ which sets the prediction problem, and a predictor network fθ^ trained on the state-action pairs of the expert trajectories. fθ takes an observation to an embedding fθ:S×AK and similarly so for fθ^:S×AK, which is analogous to the K prediction problems defined in Eq. (6). The predictor network is trained to minimize the expected mean square error by gradient descent L(s,a)=||fθ^(s,a)-fθ(s,a)||22 with respect to its parameters θ^. After training, the score of a state-action pair belonging to the support of πE is predicted by L(s,a), analogous to Eq. (7). One concern to RND is that a sufficiently powerful optimization algorithm may recover fθ perfectly, causing L(s,a) to be zero everywhere. Our experiments confirm the observations in Burda et al. (2018) that standard gradient-based methods do not behave in this undesirable way.

With the prediction error L(s,a) as the estimated score of (s,a) belonging to the support of πE , we propose a reward function that has worked well in practice,

r(s,a)=exp(-σ1L(s,a)) (8)

where σ1 is a hyperparameter. As L(s,a) is positive, r(s,a)[0,1]. We choose σ1 such that r(s,a) from expert demonstrations are mostly close to 1.

3.3 Terminal Reward Heuristic

For certain tasks, there are highly undesirable terminal states (e.g. car crashes in autonomous driving). In such contexts, We further introduce a terminal reward heuristic to penalize the RL agent from ending an episode far from the estimated support of the expert policy. Specifically, let r¯=-1Ni=1Nr(si,ai) be the average reward computed using expert trajectories, and r(sT,aT) the reward of the final state of an episode, we define

rterm={-σ2r¯ if σ3r¯>r(sT,aT)0 otherwise  (9)

where σ2,σ3 are hyperparameters. We apply the heuristic to an autonomous driving task in the experiment, and demonstrate that it corresponds nicely with dangerous driving situations and encourages RL agents to avoid them. We note that the heuristic is not needed for all other tasks considered in the experiments.

{algorithm}

[t] Random Expert Distillation

  Input: data D={(si,ai)}i=1N, Θ function models, initial policy π0.
   
  Randomly sample θΘ
  θ^=Minimize(fθ^,fθ,D)
  r()=exp(-σ1fθ^()-fθ()22)
  Compute rterm from Eq. (9)
  π=ReinforcementLearning(r,rterm,π0).
   
  Return: π.

3.4 Random Expert Distillation

We present our algorithm Random Expert Distillation (RED) in Algorithm 3.3. The algorithm computes the estimated support of the expert policy and generates a fixed reward function according to Eq. (8). The reward function is used in RL for imitation learning. As we report in the experiments, a random initial policy π0 is sufficient for the majority of tasks considered in the experiments, while the remaining tasks requires initialization via BC. We will further discuss this limitation of our method in the results.

3.5 Alternative to RND

AutoEncoder (AE) networks Vincent et al. (2008) set up a prediction problem of minθ||fθ(x)-x||22. AE behaves similarly to RND in the sense that it also yields low prediction errors for data similar to the training set. AE could be also seen as approximate support estimation in the context of Eq. (7), by replacing randomly generated fθk with identity functions on each dimension of the input. Specifically, AE sets up K prediction problems

minθ1Ni=1N(fθ(xi)-Ik(xi))2

for k=1,,K where Ik(x)=xk and K is the input size. However, as discussed in Burda et al. (2018), AE with a bottleneck layer may suffer from input stochasticity or model misspecification, which are two sources of prediction errors undesirable for estimating the similarity of input to the training data. Instead, we have found empirically that overparametrized AEs with a 2 regularization term prevent the trivial solution of an identity function, allow easier fitting of training data, and may be used in place of RND. We include AE in our experiments to demonstrate that support estimation in general appears to be a viable strategy for imitation learning.

(a) n = 5
(b) n = 10
(c) n = 50
(d) n = 100
Figure 1: True mean episodic reward during training on the simple domain, with different expert dataset sizes.

4 Experiments

We evaluate the proposed method on multiple domains. We present a toy problem to motivate the use of expert policy support estimation for imitation learning; and to highlight the behaviors of different imitation learning algorithms. We then evaluate the proposed reward function on five continuous control tasks from the Mujoco environment. Lastly, we test on an autonomous driving task with a single trajectory provided by a human driver. The code for reproducing the experiments is available online.11 1 https://github.com/RuohanW/red.git

4.1 Simple Domain

We consider a stateless task where sunif(-1,1), a discrete action space a{-1,1} and the reward function r(s,a)=as. It is clear that the expert policy to this problem is πE(s)=sign(s). Using Deep Q Learning Mnih et al. (2015) as the reinforcement learning algorithm, we compare the proposed method against AE, GAIL and GMMIL with an expert dataset of size n=5,10,50,100 respectively.

(a) GMMIL
(b) GAIL
(c) AE
(d) GMMIL fails intermittently
(e) GAIL overfits with state near 1
(f) RED
Figure 2: Estimated reward functions of the expert using different imitation learning algorithms on the simple domain. For better visualization of the reward, we use r(s,a)=1-α1L(s,a) as the reward function for AE and RED.

In Figure 1, we show the true mean episodic reward of different algorithms during training. Except for GMMIL, all other algorithms converge to the expert policy for all sizes of the dataset. In addition, RED and AE converge to the expert policy faster as the extracted reward functions recover the correct reward function nearly perfectly (Figures 1(f), 1(c)). In contrast, GAIL requires adversarial training to recover the correct reward function, a noisy process during which all actions from the learned policy, both optimal or not, are considered “wrong” by the discriminator. In fact, when the discriminator is overparametrized, it would overfit to the training data gradually and generate arbitrary reward for some region of the state-action space (Figure 1(e)). The results are consistent with observations from Brock et al. (2018) that the discriminator is overfitting, and that early stopping may be necessary to prevent training collapse. For GMMIL, we observe that the method intermittently generates wrong reward functions (Figure 1(d)), causing the performance to oscillate and converges to lower mean reward, especially when the expert data is sparse. This is likely due to the noise in estimating distribution moments from limited samples.


Table 1: Episodic reward (as provided by the environment) on Mujoco tasks by different methods evaluated over 50 runs. HalfCheetah and Ant uses BC initialization in RED and AE. We are unsuccessful with GMMIL on Ant.
Hopper HalfCheetah Reacher Walker2d Ant
GAIL 3614.22 ± 7.17 4515.70 ± 549.49 -32.37 ± 39.81 4877.98 ± 2848.37 3186.80 ± 903.57
GMMIL 3309.30 ± 26.28 3464.18 ± 476.50 -11.89 ± 5.27 2967.10 ± 702.03 -
AE 3478.31 ± 3.09 3380.74 ± 101.94 -10.91 ± 5.62 4097.61 ± 118.06 3778.61 ± 422.63
RED 3625.96 ± 4.32 3072.04 ± 84.71 -10.43 ± 5.20 4481.37 ± 20.94 3552.77 ± 348.67

4.2 Mujoco Tasks

We further evaluate RED on five continuous control tasks from the Mujoco environment: Hopper, Reacher and HalfCheetah, Walker2d and Ant. Similarly, we compare RED against AE, GAIL and GMMIL, using Trust Region Policy Optimization (TRPO) Schulman et al. (2015) in Table 1. Similar to the experiments in Ho & Ermon (2016), we consider learning with 4 trajectories of expert demonstration generated by an expert policy trained with RL. All RL algorithms terminate within 5M environment steps. We note that we were unsuccessful in the Ant task with GMMIL after an extensive search for the appropriate hyperparameters.

Figure 3: True mean episodic reward of GMMIL, GAIL, AE and RED on Hopper during training. Agents trained with RED improve faster compared to other methods.

The results suggest that support estimation of expert policies is a viable strategy for imitation learning, as the AE and RED are both able to achieve good results with the fixed reward functions constructed from support estimation of the expert policy. While RED and AE underperform on the HalfCheetah, both methods are comparable or better than GAIL and GMMIL in all other tasks. In particular, RED and AE achieve much smaller variance in performance, likely due to the fixed reward function. Consistent with the observations from the simple domain in Sec. 4.1, RED appears to achieve faster performance improvements during early training (Figure 3). We note that though the best performance AE can achieve is similar to RED, AE is much more sensitive to the random initialization of parameters, seen from the large standard deviation in Figure 3.

A limitation of our method is that HalfCheetah and Ant require a policy initialized with BC to achieve good results while GAIL and GMMIL could start with a random policy in the two tasks. We hypothesize that the evolving reward functions of GAIL and GMMIL may provide better exploration incentives to the RL agent. As GAIL is orthogonal to our method and may be used together, we leave it to future work on how to combine the benefits of both RED and GAIL to achieve more robust algorithms.

(a) Episodic reward (distance travelled without collision) on the driving task with different methods. The expert performance of 7485 corresponds with track completion without collision. Both AE and RED uses the terminal reward heuristic.

Average Std Best
GAIL 795 395 1576
GMMIL 2024 981 3624
BC 1033 474 1956
AE 4378 1726 7485
RED 4825 1552 7485
Expert 7485 0 7485
(b) The expert driver providing demonstration on the driving scenario. The scenario consists of driving through a straight track while avoiding obstacles.

4.3 Autonomous Driving Task

Lastly, we evaluate RED, AE, GAIL, GMMIL and BC on an autonomous driving task, using a single demonstration provided by a human driver. The environment consists of a straight track with obstacles placed randomly at one of the three lanes of the track (Figure 3(b)). A human demonstrator was instructed to drive through the track and avoid any obstacles, while keeping the speed around 100 km/h. We sampled the expert driving actions at 20 Hz. For the environment, we use a vector of size 24 to represent state (20 dimensions for the LIDAR reading, 3 dimensions for the relative position, and orientation of the track with respect to the car, and 1 dimension for the vehicle speed) to output the steering and the accelerator command for the vehicle. We include the terminal reward heuristic defined in Eq. 9. For evaluation, we measure the distance a learned policy is able to control the vehicle through the track without any collision with the obstacles. The task is challenging as the learner must generalize from the limited amount of training data, without explicit knowledge about the track width or the presence of obstacles. The driving task also allows us to qualitatively observe the behaviors of learned policies.

In this task, we initialize all policies with BC and use stochastic value gradient method with experience replay Heess et al. (2015) as the reinforcement learning algorithm. The algorithm is referred to as SVG(1)-ER in the original paper.

(c) Near Collision
(d) Collision
(e) Off-road Collision
(f) Off-road
Figure 4: Representative scenarios where the reward functions of AE and RED assign near zero rewards. The scenarios correspond well with various dangerous states as they are dissimilar to those from expert demonstrations.

Table 3(a) shows the average and best performance of each method evaluated over 5 runs through the same track. We note that the expert performance of 7485 corresponds with track completion without collision. The results suggest that RED achieves the best performance. For both RED and AE, the reward functions correctly identify dangerous situations, such as collision or near-collision, by assigning those states with zero rewards. Figure 4 shows a few representative states where the reward function outputs zero reward. In contrast, we observe that GAIL tends to overfit to the expert trajectory. During training, GAIL often "forgets" how to avoid the same obstacle after the discriminator update the reward function. Such behaviors prevent the learned policy from avoiding obstacles consistently. For GMMIL, we observe behaviors similar to that found in Sec. 4.1: the reward function intermittently produces problematic incentives, such as assigning positive rewards for states immediately preceding collisions.

5 Conclusion

We propose a new general framework of imitation learning via expert policy support estimation. We connect techniques such as Random Network Distillation and AutoEncoders to approximate support estimation; and introduce a method for efficiently learning a reward function suitable for imitation learning from the expert demonstrations. We have shown empirically in multiple tasks that support estimation of expert policies is a viable strategy for imitation learning, and achieves comparable or better performance compared to the state-of-art. For future works, combining different approaches of recovering the expert’s reward function appears to be a promising direction.

References

  • Abbeel & Ng (2004) Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp.  1. ACM, 2004.
  • Arjovsky & Bottou (2017) Arjovsky, M. and Bottou, L. Towards principled methods for training generative adversarial networks. stat, 1050:17, 2017.
  • Arjovsky et al. (2017) Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214–223, 2017.
  • Aronszajn (1950) Aronszajn, N. Theory of reproducing kernels. Transactions of the American mathematical society, 68(3):337–404, 1950.
  • Bansal et al. (2018) Bansal, M., Krizhevsky, A., and Ogale, A. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079, 2018.
  • Baram et al. (2017) Baram, N., Anschel, O., Caspi, I., and Mannor, S. End-to-end differentiable adversarial imitation learning. In International Conference on Machine Learning, pp. 390–399, 2017.
  • Bojarski et al. (2016) Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016.
  • Brock et al. (2018) Brock, A., Donahue, J., and Simonyan, K. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.
  • Burda et al. (2018) Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.
  • De Vito et al. (2014) De Vito, E., Rosasco, L., and Toigo, A. A universally consistent spectral estimator for the support of a distribution. Appl Comput Harmonic Anal, 37:185–217, 2014.
  • Engl et al. (1996) Engl, H. W., Hanke, M., and Neubauer, A. Regularization of inverse problems, volume 375. Springer Science & Business Media, 1996.
  • Finn et al. (2016) Finn, C., Christiano, P., Abbeel, P., and Levine, S. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852, 2016.
  • Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014.
  • Heess et al. (2015) Heess, N., Wayne, G., Silver, D., Lillicrap, T., Erez, T., and Tassa, Y. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944–2952, 2015.
  • Ho & Ermon (2016) Ho, J. and Ermon, S. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565–4573, 2016.
  • Kim & Park (2018) Kim, K.-E. and Park, H. S. Imitation learning via kernel mean embedding. AAAI, 2018.
  • Lucic et al. (2018) Lucic, M., Kurach, K., Michalski, M., Gelly, S., and Bousquet, O. Are gans created equal? a large-scale study. In Advances in neural information processing systems, pp. 698–707, 2018.
  • Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
  • Nagabandi et al. (2018) Nagabandi, A., Kahn, G., Fearing, R. S., and Levine, S. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7559–7566. IEEE, 2018.
  • Ng & Russell (2000) Ng, A. Y. and Russell, S. J. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, pp. 663–670. Morgan Kaufmann Publishers Inc., 2000.
  • Pomerleau (1991) Pomerleau, D. A. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88–97, 1991.
  • Rajeswaran et al. (2017) Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087, 2017.
  • Ross & Bagnell (2010) Ross, S. and Bagnell, D. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661–668, 2010.
  • Ross et al. (2011) Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627–635, 2011.
  • Rudi et al. (2017) Rudi, A., De Vito, E., Verri, A., and Odone, F. Regularized kernel algorithms for support estimation. Frontiers in Applied Mathematics and Statistics, 3:23, 2017.
  • Salimans et al. (2016) Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 2234–2242, 2016.
  • Schölkopf et al. (1998) Schölkopf, B., Smola, A., and Müller, K.-R. Nonlinear component analysis as a kernel eigenvalue problem. Neural computation, 10(5):1299–1319, 1998.
  • Schulman et al. (2015) Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889–1897, 2015.
  • Sun et al. (2017) Sun, W., Venkatraman, A., Gordon, G. J., Boots, B., and Bagnell, J. A. Deeply aggrevated: Differentiable imitation learning for sequential prediction. arXiv preprint arXiv:1703.01030, 2017.
  • Vincent et al. (2008) Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096–1103. ACM, 2008.
  • Ziebart et al. (2008) Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433–1438. Chicago, IL, USA, 2008.