In recent studies on model-based reinforcement learning (MBRL), incorporatinguncertainty in forward dynamics is a state-of-the-art strategy to enhancelearning performance, making MBRLs competitive to cutting-edge model freemethods, especially in simulated robotics tasks. Probabilistic ensembles withtrajectory sampling (PETS) is a leading type of MBRL, which employs Bayesianinference to dynamics modeling and model predictive control (MPC) withstochastic optimization via the cross entropy method (CEM). In this paper, wepropose a novel extension to the uncertainty-aware MBRL. Our main contributionsare twofold: Firstly, we introduce a variational inference MPC, whichreformulates various stochastic methods, including CEM, in a Bayesian fashion.Secondly, we propose a novel instance of the framework, called probabilisticaction ensembles with trajectory sampling (PaETS). As a result, our BayesianMBRL can involve multimodal uncertainties both in dynamics and optimaltrajectories. In comparison to PETS, our method consistently improvesasymptotic performance on several challenging locomotion tasks.
Quick Read (beta)
Variational Inference MPC for
Bayesian Model-based Reinforcement Learning
In recent studies on model-based reinforcement learning (MBRL), incorporating uncertainty in forward dynamics is a state-of-the-art strategy to enhance learning performance, making MBRLs competitive to cutting-edge model-free methods, especially in simulated robotics tasks. Probabilistic ensembles with trajectory sampling (PETS) is a leading type of MBRL, which employs Bayesian inference to dynamics modeling and model predictive control (MPC) with stochastic optimization via the cross entropy method (CEM). In this paper, we propose a novel extension to the uncertainty-aware MBRL. Our main contributions are twofold: Firstly, we introduce a variational inference MPC (VI-MPC), which reformulates various stochastic methods, including CEM, in a Bayesian fashion. Secondly, we propose a novel instance of the framework, called probabilistic action ensembles with trajectory sampling (PaETS). As a result, our Bayesian MBRL can involve multimodal uncertainties both in dynamics and optimal trajectories. In comparison to PETS, our method consistently improves asymptotic performance on several challenging locomotion tasks.
|Panasonic Corp., Japan|
|Ritsumeikan Univ. & Panasonic Corp., Japan|
Keywords: model predictive control, variational inference, model-based reinforcement learning
Model predictive control (MPC) is a powerful and accepted technology for advanced control systems such as manufacturing processes [vargas2000multilayer], HVAC systems [afram2014theory], power electronics [vazquez2014model], autonomous vehicles [paden2016survey], and humanoids [kuindersma2016optimization]. MPC utilizes the specified models of system dynamics to predict future states and rewards (or costs) to plan future actions that maximize the total reward over the predicted trajectories. Especially for industrial applications, the clear explainability of such a decision-making process is advantageous. Furthermore, in some tasks (e.g., games) [silver2016mastering], planning-based policies of this nature could outperform reactive-policies (e.g., full neural network policies).
Model-based reinforcement learning (MBRL) methods that employ expressive function approximators (e.g., deep neural networks: DNNs) [deisenroth2011pilco, williams2017information, nagabandi2018neural] present appealing approaches for MPC. The main difficulty in introducing MPC to practical systems is specifying the forward dynamics models of target systems. However, accurate system identification is challenging in many advanced applications. Take robotics for example, where robots encounter floors and walls, and must be able to manipulate some objects, making the dynamics highly non-linear. The main objective of MBRL is to train approximators of complex dynamics through experiences in real systems. The general procedure of MBRL is summarized as; (1. training-step) train the approximate model with a given training dataset, then (2. test-step) execute the actions (or policies) optimized with the dynamics model in a real environment and augment the dataset with the observed results. The above training and test steps are iteratively conducted to collect sufficient and diverse data so as to achieve the desired performance.
One feature of MBRL is its considerable sample efficiency compared to model-free reinforcement learning (MFRL), which directly trains policies through experiences. In other words, MBRL requires much less test time in real environments. In addition, MBRL benefits from the generalizability of the trained model, which can be easily applied to new tasks in the same system. However, the asymptotic performance of MBRL is generally inferior to that of model-free methods. This discrepancy is primarily due to the overfitting of dynamics models to the few data available during initial MBRL steps, which is called the model-bias problem [deisenroth2011pilco]. Several studies have demonstrated that incorporating uncertainty in dynamics models can alleviate this issue. The uncertainty-aware modeling is realized by Bayesian inference employing a Gaussian Process [deisenroth2011pilco], dropout as variational inference [gal2016dropout, gal2017concrete, kahn2017uncertainty], or neural network ensembles [chua2018deep, kurutach2018model, clavera2018model].
Probabilistic ensembles with trajectory sampling (PETS) [chua2018deep] is one type of uncertainty-aware MBRL. As an MPC-oriented MBRL method, PETS conducts trajectory optimization via the cross entropy method (CEM) [botev2013cross] by using trajectories probabilistically sampled from the ensemble networks. Experiments have demonstrated that PETS can achieve competitive performance over state-of-the-art MFRL methods like Soft Actor Critic (SAC) [haarnoja2018soft], while yielding much higher sample efficiency. Since our primary interest is MPC and its application to practical systems, this paper mainly focuses on PETS and treats this method as a strong baseline.
Considering the success of probabilistic dynamics modeling, incorporating uncertainty in optimal trajectories appears very promising for MBRL. However, an optimization scheme that can utilize uncertainty has not yet been discussed. Although several stochastic approaches, including CEM, model predictive path integral (MPPI) [williams2016aggressive, williams2017information], covariance matrix adaptation evolution strategy (CMA-ES) [hansen2003reducing], and proportional CEM (Prop-CEM) [goschin2013cross], have been proposed, they are not uncertainty-aware and tend to underestimate uncertainty. In addition, although their optimization procedures are very similar, they have been independently derived. Consequently, theoretical relations among these methods are unclear, preventing us from systematically understanding and reformulating them to be uncertainty-aware in a Bayesian fashion.
Motivated by these, in this paper, we propose a novel MPC concept for Bayesian MBRL. The organization and contributions of this paper are summarized as follows. (1) In Sec. 3, we introduce a novel MPC framework, variational inference MPC (VI-MPC), which generalizes and reformulates various stochastic MPC methods in a Bayesian fashion. The key observations for deriving this framework are organized in Sec. 2, where we point out that general stochastic optimization methods can be regarded as the moment matching of the optimal trajectory posterior, which appear in a Bayesian MBRL formulation. (2) In Sec. 4, we propose a novel instance of the framework, called probabilistic action ensembles with trajectory sampling (PaETS). Toy task examples and the concept of our method are exhibited in Fig. 1. (3) In Sec. 5, we demonstrate that our method consistently outperforms PETS via experiments with challenging locomotion tasks in the MuJoCo physics simulator [todorov2012mujoco].
2 Model-based Reinforcement Learning as Bayesian Inference
In this section, we describe MBRL as a Bayesian inference problem using control as inference framework [levine2018reinforcement]. Fig. 2 displays the graphical model for the formulation, with which an MBRL procedure can be re-written in a Bayesian fashion: (1. training-step) do inference of . (2. test-step) do inference of , then, sample actions from the posterior and execute the actions in a real environment. We denote a trajectory as , where and respectively represent state and action. Given a state-action pair at time , the next state can be predicted by a forward-dynamics model parameterized with . The posterior of is inferred from training dataset , where consists of states and actions observed during the test step. To formulate optimal control as inference, we auxiliarly introduce a binary random variable to represent the optimality of (, ). Given , trajectory optimization can be expressed as an inference problem:
where uninformative action prior (i.e., : uniform distribution) is supposed. For readability, is simply denoted as . For the same reason, we omit the subscripts of sequences , . In the remainder of the paper, this simplified notation is employed. In Sec. 2.1–2.2, we review how these inference problems have been approximately handled in previous works.
2.1 Inference of Forward-dynamics Posterior
Given a sufficiently parameterized expressive model, i.e., DNNs, one of the most practical and promising schemes for approximating the posterior is to utilize neural network ensembles [chua2018deep, kurutach2018model, clavera2018model]. This process approximates the posterior as a set of particles , where is Dirac delta function and is the number of networks. Each particle is independently trained by stochastic gradient descent so as to (sub-)optimize . Although this approximation is incompletely Bayesian, this scheme has several useful features. First, we can simply implement this process in standard deep learning frameworks. Furthermore, the ensemble model successfully involves multimodal uncertainty in the exact posterior.
Another possible way to infer is dropout as variational inference [gal2016dropout, gal2017concrete, kahn2017uncertainty], which approximates as a Gaussian distribution . It is proofed that the variational inference problem: approximately equivalent to training networks with dropout, where denote Kullback-Leibler (KL) divergence. Although this scheme is also simple and theoretically supported, approximation by a single Gaussian distribution tends to underestimate uncertainty (or multimodality) in the posterior. To remedy this problem, -divergence dropout has been proposed [li2017dropout], which replaces KL-divergence to -divergence so as to prevent from overfitting a single mode. However, as long as is Gaussian, the multimodality cannot be managed well.
In our preliminary experiments of MBRL, we have tested the above two schemes and observed that the ensemble performs much better than (-)dropout. This result provides us with the insight that capturing multimodality in the posterior has crucial effects in MBRL literature. Therefore, in this paper, we also employ this ensemble scheme to approximate in the same way as our baseline: PETS [chua2018deep]. In Sec. 4, we also attempt to incorporate multimodality in the posterior .
2.2 Moment Matching of Trajectory Posterior
This section clarifies the connection between trajectory optimization and the posterior approximation problem. The key observation delineated here is that several MPC methods, including CEM used in PETS and MPPI, can be regarded as the moment matching of the posterior.
Given an inferred model posterior , we can sample trajectories from (1).11 1 Trajectory sampling methods with have been discussed and experimented in [chua2018deep]. In this paper, we employ the TS1 method suggested in the reference (see 3–6 in Alg. 1). Let us approximate the action posterior with a Gaussian distribution . The mean of posterior action sequence can be estimated by moment matching:
Eq. (2) can be viewed as a weighted average where each sampled action is weighted by the likelihood of optimality . In the same way, we can also estimate the variance of the posterior .
In practice, sampling from uniform distribution is quite inefficient and requires almost infinite samples. Hence, let us consider iteratively estimating the parameters by incorporating importance sampling. Let , be the estimated parameters at iteration ; we can rearrange (2) as
It is worth noting that a similar iterative law can also be derived by solving the optimization problem by mirror descent [miyashita2018mirror, okada2018acceleration]. To connect this inference problem to trajectory optimization, we define the optimality likelihood with trajectory reward and a monotonically increasing function , as . If we define the same as [levine2018reinforcement, piche2018probabilistic], an optimization algorithm similar to MPPI [williams2016aggressive, williams2017information, okada2018acceleration] is recovered. As summarized in Table 1, other similarities to well-known optimization algorithms, including CEM, can be observed with different optimality definitions. 22 2 We implicitly assume the existence of step-wise likelihood corresponding to each definition. Since another graphical model with a single unified optimality can be defined, the existence is not critical.
|MPPI [williams2016aggressive]||CEM [botev2013cross]||Prop-CEM [goschin2013cross]||CMA-ES [hansen2003reducing]|
There is a discrepancy between (4) and the CEM implementation in [chua2018deep]; in which is used instead of . Since is a convex function, Jensen’s inequality holds in this case, thus . The equality holds when is constant, implying that for low-variance and for high-variance (or more uncertain) . Namely, underestimates the optimality likelihood if generates uncertain trajectories. Since we have experimentally observed that this filtering effect of demonstrates higher optimization performance than (see Sec. A), this paper heuristically employs the use of .
In practice, expectation operators should be implemented on digital computers through the Monte Carlo integration with sampled actions and trajectories for each action: and .
3 Variational Inference MPC: From Moment Matching to Inference
Given uncertainty in a dynamics model, it is natural to suppose that optimal trajectories are also uncertain. However, as exhibited in the previous section, PETS employs the moment matching of the trajectory posterior, ignoring almost uncertainty in optimal trajectories. In this section, we newly introduce a variational inference MPC (VI-MPC) framework to formulate MBRL as fully Bayesian and involve uncertainty both in the dynamics and optimalities.
Let us consider a variational inference problem: . We assume the variational distribution is decomposed to ; hence, we introduce as a posterior, which takes the similar decomposable form as . This assumption forces optimal state transitions to be controlled only by [levine2018reinforcement]. As shown in Sec. B.1, this inference problem can be transformed to the maximization problem: . A notable property is that this objective has an entropy regularization term , which encourages to have broader shape to capture more uncertainty. For the sake of convenience, we introduce a tunable hyperparameter to the optimality likelihood . Then the above objective can be transformed as . By applying mirror descent [bubeck2015convex] to this optimization problem, we can derive an update law for (see Sec. C for the detailed derivation):
where , are hyperparameters and is absorbed into them. is inverted step-size to control optimization speed and is the weight of the entropy regularization term .
Eq. (5) suggests a novel and general MPC framework, which we call variational inference MPC (VI-MPC). To realize a specific VI-MPC method, we specify the following parameters: (1) optimality definition (or ; see Table 1), (2) variational distribution model , and (3) entropy regularization or . We did not include into the specifications since it is highly dependent on the optimality definition (see Sec. G). In this paper, we describe the above specifications as VIMPC(<optimality_def>, <variational_dist>, <max_ent>). For example, we respectively express vanilla CEM and MPPI as VIMPC(‘CEM’, ‘Gaussian’, False) and VIMPC(‘MPPI’, ‘Gaussian’, False). In Sec. 4, we propose a new instance of VI-MPC to incorporate multimodal uncertainty in the posterior.
4 Probabilistic Action Ensembles with Trajectory Sampling
As reviewed in Sec. 2.1, previous methods have successfully involved multimodality in with network ensembles. If this multimodality in is given, other distributions depending on , including , would also be multimodal. In other words, there are various possible optimal trajectories (or actions) like Fig. 1. It is obvious that VIMPC(*, ‘Gaussian’, *) will still easily fail to capture multimodality because of overfitting to a single mode. Inspired by the success of the ensemble approach for dynamics modeling, we propose a novel VI-MPC method that introduces action ensembles with a Gaussian mixture model (GMM), i.e., VIMPC(*, ‘GMM(M=*)’, *), which we call PaETS (Probabilistic Action Ensembles with Trajectory Sampling).
PaETS defines the variational distribution as
where and is the number of components of the mixture model. Now, we derive the iteration scheme to update the parameters of GMM. At first, drawing samples from , we approximate as a discretized distribution (or a set of particles):
where . Just after sampling, the weight of each particle is uniform: . By substituting this approximated distribution to (5), the update law for the particle weights is derived as
Then we estimate , which maximizes the observation probability of the weighted particles:
By taking the derivative and borrowing the concept of the EM algorithm [bilmes1998gentle], we get the update laws of which take the weight-average form like (4) (see Sec. D for the complete definition):
In summary, PaETS and the MPC utilizing it are respectively described in Algs. 1 and 2, where is the number of iterations for optimization and is the length of the task episode. At in Alg. 2, s are initialized independently at random. At , s and s are reset to be initial values, encouraging exploration for the next time-step and preventing from degenerating to a single mode. If we set , these procedures are almost equivalent to those of PETS. The use of GMM () does not increase computational complexity significantly (see Sec. F). \SetCommentStymycommfont
5.1 Comparison to State-of-the-art Methods
The main objective of this experiment is to demonstrate that PaETS has advantages over the state-of-the-art MBRL baseline: PETS [chua2018deep]. In this experiment, PaETS and PETS (or vanilla CEM) were implemented using our same codebase with different parameters, i.e., VIMPC(‘CEM’, ‘GMM(M=5)’, True) for PaETS, and VIMPC(‘CEM’, ‘GMM(M=1)’, False) for PETS. We also evaluated another MBRL baseline with MPPI [williams2017information], realized as VIMPC(‘MPPI’, ‘GMM(M=1)’, False). These above methods share the settings for inference (training of network ensembles). The state-of-the-art MFRL method SAC [haarnoja2018soft], was also evaluated to compare asymptotic performance.33 3 We used the open-source code: https://github.com/pranz24/pytorch-soft-actor-critic Fig. 3 illustrates the simulated locomotion tasks evaluated in this experiment, which are complex and challenging due to their high non-linearity. All the tasks, except for HalfCheetah, were not evaluated in the original PETS paper [chua2018deep]. Other details about our implementation and experimental settings are described in Sec. G and Sec. H. Fig. 4 presents the experimental results, in which PaETS consistently exhibits better asymptotic performance than that of the MBRL baselines. In addition, PaETS outperforms or is comparable to SAC while requiring significantly fewer samples (about x10 more sample efficient).
5.2 Ablation Study
This experiment clarifies which component of PaETS (GMM and entropy-regularization) contributed to the overall improvement. Fig. 5 expresses the results of this ablation study and Welch’s -test for some selected representative pairs. From this figure, one can observe that the use of GMM () significantly improves performance. The effect of the regularization () is relatively small, but not negligible. In certain tasks, setting to particular values could improve the performance. In the case of , the regularization sheds light on actions sampled from low , thus encouraging to be multimodal. In some tasks which requires rather delicate controls (e.g., Hopper, Walker2d), the effect of seems less significant. Fig. 6 examines sensitivity with the number of mixture components , for which achieved the highest performance. If infinite or enough samples are given (), it would be reasonable to set to be large enough to capture multimodality. However, in practice, is finite and could be small enough due to computational constraints. In this case, larger makes it difficult to approximate as a set of particles , resulting in degradation of the optimization performance.
6 Related Work
Dynamics Posterior Inference Recent MBRL methods, MB-MPO (Model-Based Meta-Policy-Optimization) [clavera2018model] and ME-TRPO (Model Ensemble Trust Region Optimization) [kurutach2018model], also employ network ensembles to model dynamics, but they utilize the ensembles differently than we do: to train policy networks, not MPC.
Trajectory Optimization Sequential Monte-Carlo based MPC, described as VIMPC(*, ‘Particles’, False), has been introduced in [kantas2009sequential], but it requires well-designed proposal distribution to sample particles for the next iteration . Another particle-based method has been derived [piche2018probabilistic] by utilizing the control as inference framework. However, this method relies on not only a dynamics model, but also policy and value functions to manage particles, so MFRL methods must be incorporated.
Recent studies have demonstrated that entropy regularization is a promising strategy in policy training [abdolmaleki2015model, abdolmaleki2017deriving, haarnoja2017reinforcement, haarnoja2018soft]. However, to the best of our knowledge, the introduction of entropy regularization to MPC is novel along with explicit multimodal expression to successfully realize their synergistic effect.
Ref. [wagener2019online] also systematically organizes the stochastic MPC methods from the perspective of online learning, but uncertainty-aware discussions from a Bayesian viewpoint are not conducted.
Bayesian Reformulation Ref. [jeon2018bayesian] proposes a novel approach to generative adversarial imitation learning (GAIL) [ho2016generative], which reformulates general GAIL in a Bayesian fashion and utilizes ensembles to infer discriminator posteriors. Another Bayesian reformulation of GAIL integrates imitation and reinforcement learning by introducing another optimality (i.e., imitation optimality ) [kinose2019integration].
7 Conclusion & Discussions
This paper introduces a novel VI-MPC framework that systematically generalizes and reformulates various stochastic MPC methods in a Bayesian fashion. We also devise a novel instance of this framework, called PaETS, which can successfully incorporate multimodal uncertainty in optimal trajectories. By combining our method and the recent uncertainty-aware dynamics modeling with neural network ensembles, our Bayesian MBRL is able to involve multimodalities both in dynamics and optimalities. In addition, our method is a quite simple extension of general stochastic methods and requires no significant additional computational complexity. Our experiments demonstrate that PaETS can improve asymptotic performance compared to the leading MBRL baseline PETS, and thus substantially enhances MBRL potential to be more competitive to the state-of-the-art MFRL.
Considering the simplicity and generalizability of VI-MPC and PaETS, we expect that our concept is applicable to a variety of tasks, such as traditional MPC with deterministic dynamics and advanced MPC with latent dynamics from pixels by Deep Planning Network [hafner2018learning]. By introducing a categorical mixture model as a variational distribution, application to combinational optimizations is also feasible. In fact, our ongoing work includes experiments of discrete MPC for a practical system.
A question that remains is how to determine VI-MPC specifications. As implied in Fig. 4, the best optimality definition could be task dependent (e.g., MPPI outperformed vanilla CEM in the Ant but not in other tasks). The regularization weight also has task dependency as shown in Fig. 5. It would be challenging but interesting future work to add the parameters to the graphical model in Fig. 2 as latent variables to infer promising parameters along with optimal trajectories, like infinite GMM [rasmussen2000infinite]. Another appealing endeavor for future work is to introduce the concept of parallel tempering [brooks2011handbook] in Markov Chain Monte Carlo. By adaptively varying different temperatures ( in our case) of ensemble actions, we can expect the ensemble diversity to improve.
We thank Vishwajeet Singh, Hiroki Nakamura and Akira Kinose for their cooperation in this study during their student-internship periods. Most of the experiments were conducted in ABCI (AI Bridging Cloud Infrastructure), built by the National Institute of Advanced Industrial Science and Technology, Japan.
Appendix A Comparison Between and
We evaluated the impact of and on the optimization performance of (vanilla) CEM and MPPI, the results of which are summarized in Table 2, where gained much higher rewards than .
Appendix B Derivations
B.1 Derivation of the Variational Inference Objective
By using the assumption of , the KL-divergence can be transformed as
Appendix C Derivation of (5)
In this section, we simply denote as and as for readability. Let us consider the optimization problem:
By applying mirror descent [bubeck2015convex], the iterative update law of is given as
where is the inner-product operator, is a hyper-parameter related to the step-size, and is the Lagrange multiplier for the constraint . The arguments in the operator can be rearranged as
where, we used the relations:
The integrand of (16) can be organized as
Integrating the above equation yields,
By minimizing this equation, we get:
The Lagrange multiplier can be removed using the constraint :
Marginalizing ), we finally obtain:
In (5), we replaced , .
Appendix D Complete Definition of PaETS
Appendix E Optimization of Toy Objective Function by PaETS
Fig. 7 illustrates how PaETS optimizes in a toy multimodal objective function.
Appendix F Computational Complexity
The main computational bottleneck of PaETS (and PETS) is the execution of 3–6 in Alg. 1, in which total trajectories must be sampled. In our experiment, and were respectively set as , as in [chua2018deep]. Compared to PETS, PaETS requires additional procedures like action sampling from GMM (2) and GMM parameter update (9). However, these additional procedures are easily parallelizable on GPUs, and their computation times are much shorter than the above mentioned bottleneck. In the experiments with our early prototype in TensorFlow, it took about 57 ms for and 55 ms for (equivalent to PETS) to execute one iteration of the for-loop in Alg. 1 on a single NVIDIA RTX2080 GPU. The above execution time does not meet the real-time constraints (e.g., 30 Hz). However, considering the success of the real-time implementation of MPPI in [williams2016aggressive, williams2017information], we believe real-time implantation of our method is feasible with optimized implementation using compiled language, low-level GPU APIs, and thorough tuning of hyperparameters (e.g., , and DNN complexity).
Appendix G Implementation Notes
Cross Entropy Method
It is general technique to adaptively determine in Table 1 so that only the top- samples satisfies the threshold condition. We employ this technique and the eliteness ratio is set to be . has no effect on CEM optimization since takes binary values.
Reward normalization heuristics, as suggested in [theodorou2010generalized], were also introduced for our MPPI implementation as
where . was set to be as also suggested in [theodorou2010generalized].
The value of is very sensitive to task settings, especially for the dimensionalities of action spaces. To make insensitive, we introduced the following normalization trick inspired by the above heuristics. First, we rearrange (8) as
Then, we replace to normalized one:
By applying these heuristics, the range of entropy bonus is limited to , where the action with the lowest probability among samples gains the highest entropy bonus of .
Appendix H Experimental Setup
We used MuJoCo tasks modified from standard OpenAI Gym tasks.44 4 https://github.com/openai/gym Table 3 summarizes the task settings, where , and respectively denote the velocity, orientation angle, and height of the agents. Penalty functions , are newly introduced to encourage the agents to move forward in the proper form. Instead, done flags used originally for early task stopping are removed. , are defined as
We modified the range of actions (i.e., torques) from to to exaggerate uncertainties in the optimal trajectory posteriors.
Table 4 summarizes the shared parameter settings for MBRL (PaETS, PETS, and MPPI). For SAC, we used the default parameters from the original codebase.
|: prediction horizon||30||30||60||45|
|: weight of entropy regularizer||0.5||0.25||0.5||0.5|
|: # sampled actions||500|
|: # trajectories for each action||20|
|: # optimization-iterations||5|
|: # episode length||1000|
|: # neural networks||5|
|hidden nodes||(200, 200, 200, 200)|