Probabilistic Graphical Models and Variational Inference play an importantrole in recent advances in Deep Reinforcement Learning. As a self-inclusivetutorial survey, this article illustrates basic concepts of reinforcementlearning with Probabilistic Graphical Models and offers derivation of somebasic formula as a recap. Reviews and comparisons on recent advances in deepreinforcement learning are made from various aspects. We offer ProbabilisticGraphical Models, detailed explanation and derivation to several use cases ofGraphical Model and Variational Inference, which serves as a complementarymaterial on top of the original contributions.
Quick Read (beta)
Tutorial and Survey on Probabilistic Graphical Model and Variational Inference in Deep Reinforcement Learning
Probabilistic Graphical Models and Variational Inference play an important role in recent advances in Deep Reinforcement Learning. As a self-inclusive tutorial survey, this article illustrates basic concepts of reinforcement learning with Probabilistic Graphical Models and offers derivation of some basic formula as a recap. Reviews and comparisons on recent advances in deep reinforcement learning are made from various aspects. We offer Probabilistic Graphical Models, detailed explanation and derivation to several use cases of Graphical Model and Variational Inference, which serves as a complementary material on top of the original contributions.
Reinforcement Learning, powered by Deep Neural Networks, has gaining increasing attention recently due to its great success in complicated tasks like games  and robot locomotion , as well as optimization tasks like Automatic Machine Learning . Despite the existing survey  in Deep Reinforcement Learning, this paper, however, focuses on the application of Probabilistic Graphical Model and Variational Inference, especially Amortized Variational Inference in Deep Reinforcement Learning. Specifically, we make the following contributions:
We cover a taxonomy of Graphical Model and Variational Inference  used in Deep Reinforcement Learning and give detailed derivation for many of the critical equations, which is either not given in the original contributions [30, 15, 11], or in a slightly different way. This makes the paper in a relative standalone position to serve as a self-inclusive tutorial from beginner to advanced.
I-A Organization of the paper
In section I-B, we first introduce the fundamentals of Probabilistic Graphical Models and Variational Inference, then we review the basics about reinforcement learning by connecting probabilistic graphical models (PGM) in section II-A, as well as the basics and a incomplete overview about deep reinforcement learning, accompanied with a comparison of different methods in section II-E. In section III, we discuss how undirected graph could be used in modeling both the value function and the policy, which works well on high dimensional discrete state and action spaces. In section IV, we introduce the directed acyclic graph framework on how to treat the policy as posterior on actions, while adding many proofs that does not exist in the original contributions. In section V, we introduce works on how to use variational inference to approximate the environment model, while adding graphical models and proofs which does not exist in the original contributions.
I-B Prerequisite on Probabilistic Graphical Models and Variational Inference, Terminologies and Conventions
Directed Acyclic Graphs (DAG)  as a PGM offers an instinctive way of defining factorized join distributions of Random Variables (RV) by assuming the conditional independence  through d-separation . We use capital letter to denote a RV, while using the lower case letter to represent the realization of corresponding RV. To avoid symbol collision of using to represent advantage in many RL literature, we use explicitly to represent action. We use to represent is conditionally independent from , given , which is equivalent to write or .
Variational Inference (VI) approximates intractable posterior distribution, specified in a probabilistic graphical model usually, with a variational proposal posterior distribution, by optimizing the Evidence Lower Bound (ELBO) , which assigns the values of latent unobservables at the same time. Variational Inference is widely used in Deep Learning Community like variational resampling . VI is also used in approximating the posterior on the weights distribution of neural networks for Thompson Sampling to tackle the exploration-exploitationtrade off in bandit problems , as well as approximating on the activations distribution like Variational AutoEncoder using Amortized VI .
II Reinforcement Learning and Deep Reinforcement Learning
II-A Basics about Reinforcement Learning with graphical model
II-A1 RL Concepts, Terminology and Convention
As shown in Figure 1, Reinforcement Learning (RL) involves optimizing the behavior of an agent via interaction with the environment. At time , the agent lives on state , By executing an action according to a policy  , the agent jumps to another state , while receiving a reward . Let discount factor decides how much the immediate reward is favored compared to longer term return, with which one could also allow tractability in infinite horizon reinforcement learning , as well as reducing variance in Monte Carlo setting . The goal is to maximize the accumulated rewards, which is usually termed return in RL literature.
For simplicity, we interchangeably use two conventions whenever convenient: Suppose an episode last from , with correspond to continuous non-episodic reinforcement learning. We use another convention of by assuming when episode ends, the agent stays at a self absorbing state with a null action, while receiving null reward.
By unrolling Figure 1, we get a sequence of state, action and reward tuples in an episode, which is coined trajectory [33, 6]. Figure 2 illustrates part of a trajectory in one rollout. The state space and action space , which can be either discrete or continuous and multi-dimensional, are each represented with one continuous dimension in Figure 2 and plotted in an orthogonal way with different colors, while we use the thickness of the plate to represent the reward space .
II-A2 DAGs for (Partially Observed ) Markov Decision Process
Reinforcement Learning is a stochastic decision process, which usually comes with three folds of uncertainty. That is, under a particular stochastic policy characterized by , within a particular environment characterized by state transition probability and reward distribution function , a learning agent could observe different trajectories with different unrolling realizations. This is usually modeled as a Markov Decision Process , with its graphical model shown in Figure 3, where we could define a joint probability distribution over the trajectory of state , action and reward RVs. In Figure 3, we use dashed arrows connecting state and action to represent the policy, upon fixed policy, we have the trajectory likelihood in Equation (1)
Upon observation of a state in Figure 3, the action at the time step in question is conditionally independent with the state and action history , which could be denoted as .
A more realistic model, however, is the Partially Observable Markov Decision process , with its Directed Acyclic Graph  representation shown in Figure 4, where the agent could only observe the state partially of getting through a non invertible function of the latent state and the previous action , as indicated the Figure by , while the distributions on other edges are omitted since they are the same as in Figure 3. Under the graph specification of Figure 4, the observable is no longer Markov, but depends on the whole history. However, by introducing a probability distribution over the hidden state , with , which is termed belief state , where state takes value in range .
II-B Value Function, Bellman Equation, Policy Iteration
where takes value from , taking value from , and we have used the and in the subscript of the expectation operation to represent the probability distribution of the policy and the environment (including transition probability and reward probability) respectively. State action value function  is defined in Equation (4),
, where in Equation (5), its relationship to the state value function is stated.
Define optimal policy  to be
Taking the optimal policy into the Bellman Equation in Equation (3), we have
Taking the optimal policy into Equation (4), we have
For learning the optimal policy and value function, General Policy Iteration  can be conducted, as shown in Figure 5, where a contracting process  is drawn. Starting from initial policy , the corresponding value function could be estimated, which could result in improved policy by greedy maximization over actions. The contracting process is supposed to converge to the optimal policy .
As theoretically fundamentals of learning algorithms, Dynamic programming and Monte Carlo learning serve as two extremeties of complete knowledge of environment and complete model free , while time difference learning  is more ubiquitously used, like a bridge connecting the two extremities. Time difference learning is based on the Bellman update error .
II-C Policy Gradient and Actor Critic
Reinforcement Learning could be viewed as a functional optimization process. We could define an objective function over a policy , as a functional, characterized by parameter , which could correspond to the neural network weights, for example.
Suppose all episodes start from an auxiliary initial state , which with probability , jumps to different state without reward. characterizes the initial state distribution which only depends on the environment. Let represent the expected number of steps spent on state , which can be calculated by summing up the discounted probability of entering state with steps from auxiliary state , as stated in Equation (12), which can be thought of as the expectation of the R.V. conditional on state .
In Equation (13), the quantity is calculated by either directly starting from state , which correspond to in Equation (12), or entering state from state with one step, corresponding to in Equation (12).
For an arbitrary state , using and to represent subsequent states as dummy index, we have
, where represent the state of steps after and already includes integration of intermediate state before reaching state .
Let objective function with respect to policy be defined to be the value function starting from auxiliary state as in Equation (19).
, where is the relative occupancy of state . The integration of with respect to and in the nominator with respect to in Equation (23) is replaced with expectation with respect to interaction with the environment in Equation (24), and is replaced by an estimator , which is usually . Functional approximation to should be compatible with policy parameterization in the sense that .
The policy gradient could be augmented to include zero gradient baseline , with respect to objective function in Equation (23), as a function of state , which does not include parameters for policy , since . To reduce variance of the gradient, the baseline is usually chosen to be the state value function estimator to smooth out the variation of at each state, while is updated in a Monte Carlo way by comparing with .
The actor-critic algorithm  decomposes to be , so bootstrap is used instead of Monte Carlo.
II-D Policy Optimization and Trust Region Method
Based on Policy Advantage  , a local approximation could be defined in Equation (26), which could form a surrogate function in Equation (27) that minorizes at , where is the maximum KL divergence, so MM algorithm could be used to improve the policy, leading to the trust region method .
II-E Basics of Deep Reinforcement Learning
Deep Q learning  makes a breakthrough in using neural network as functional approximator on complicated tasks. It solves the experience correlation problem by using a reply memory and the instability of the target problem with a frozen target network. Specifically, the reinforcement learning is transformed in a supervised learning task by fitting on the target from the replay memory with state as input. However, the target can get drifted easily which leads to unstable learning. In , a target network is used to provide a stable target for the updating network to be learned before getting updated occasionally. Double Deep Q learning , however, solves the problem by having two Q network and update the parameters in a alternating way.
II-E1 On Policy methods
A3C  stands out in the asynchronous methods in deep learning  which can be run in parallel on a single multi-core CPU. Trust Region Policy Optimization  and Proximal Policy Optimization  assimilates the natural policy gradient, which use a local approximation to the expected return. The local approximation could serve as a lower bound for the expected return, which can be optimized safely subject to the KL divergence constraint between two subsequent policies, while in practice, the constraint is relaxed to be a regularization.
II-E2 Off Policy methods
II-E3 Goal based Reinforcement Learning
In robot manipulation tasks, the goal could be represented with state in some cases . Universal Value Function Approximator (UVFA)  incorporate the goal into the deep neural network, which let the neural network functional approximator also generalize to goal changes in tasks, similar to Recommendation System. Work of this direction include [1, 33], for example.
II-E4 Replay Memory Manipulation based Method
Replay memory is a critical component in Deep Reinforcement Learning, which solves the problem of correlated transition in one episode. Beyond the uniform sampling of replay memory in Deep Q Network , Prioritized Experience Replay  improves the performance by giving priority to those transitions with bigger TD error, while Hindsight Experience Replay (HER)  manipulate the replay memory with changing goals to transition so as to change reward to promote exploration. Maximum entropy regularized multi goal reinforcement learning  gives priority to those rarely occurred trajectory in sampling, which has been shown to improve over HER .
II-E5 Exploration with sparse reward
In complicated real environment, an agent has to explore for a long trajectory before it can get any reward as feedback. Due to lack to enough rewards, traditional Reinforcement Learning methods performs poorly, which lead to a lot of contributions in the sufficient exploration methods. The methods using graphical model and variational method we introduce later each use different mechanisms to explore the environments. In the following sections, we give detailed explanation on how graphical model and variational inference could be used to model and optimize the reinforcement learning process with each category a different section. Together with the methods mentioned above, we make a comparison of them in Table I.
II-E6 Surrogate optimization
Like surrogate model used in Bayesian Optimization , lower bound surrogate is also used in Reinforcement Learning, including Evidence Lower Bound (ELBO) based methods introduced below and Trust Region Policy Optimization (TRPO) .
III Policy and value function with undirected graphs
We first discuss the application of undirected graphs in deep reinforcement learning, where we use deep belief network here. Rather than modeling conditional distribution, as in directed acyclic graphs, undirected graphs model joint distribution of variables in question and focus on cliques  with free energy associated with it, which could be used to model the value function in reinforcement learning. Restricted Boltzman Machine has nice property of tractable factorized posterior distribution over the latent variable conditioned on observables, instead of having to do gibbs sampling in general Boltzman Machine.
In , the authors use Restricted Bolzman Machine to deal with MDPs of large state and action spaces, by modeling the state-action value function with the negative free energy of the graph, where free energy of the graph could be easily calculated through the product of expert . Specifically, the visible states of the Restricted Bolzmann Machine  consists of both state and action binary variables, as shown in Figure 6, where the hidden nodes consist of binary variables, while state variable are dark colored to represent it can be observed and action are light colored to represent it need to be sampled. Together with the auxilliary hidden variables, the undirected graph defines a joint probability distribution over state and action pairs, which defines a stochastic policy network that could sample actions out for on policy learning. Since it is pretty easy to calculate the derivative of the free energy with respect to the coefficient of the network, one could use temporal difference learning to update the coefficients in the network. Thanks to properties of Boltzmann Machine, the conditional distribution of action over state is still Boltzmann distributed, governed by the free energy, by adjusting the temperature, one could also change between different exploration strength.
The conditional distribution of actions under state could serve as the policy, which is
, where is the partition function  and we use the negative free energy to approximate the state action value function. Upon the state value function in Equation (28) is learned as a critic , such that its associated policy is defined, MCMC sampling  could be used to sample actions, as an actor . With the sampled actions, time difference learning method like SARSAR , could be carried out to update the state value function estimation. Such an on-policy process has been shown to be empirically effective in the large state actions spaces .
IV Variational Inference on Policies
IV-A policy as ”optimal” posterior
The Boltzmann Machine defined Product of Expert Model in  works well for large state and action spaces, but are limited to discrete specifically binary state and action variables. For continuous state and action spaces, in , the author proposed deep energy based models with Directed Acyclic Graphs (DAG) , which we re-organize in a different form in Figure 7 with annotations added. The difference with respect to Figure 3 is that, in Figure 7, the reward is not explicit expressed in the directed graphical model. Instead, an auxilliary binary Observable is used to define whether the corresponding action at the current step is optimal or not. The conditional probability of the action being optimal is , which connects conditional optimality with the amount of award received by encouraging the agent to take highly rewarded actions in an exponential manner. Note that the reward here must be negative to ensure the validity of probability, which does not hurt generality since reward range can be translated .
By doing so, the author is forcing a form of functional expression on top of the conditional independence structure of the graph by assigning a likelihood. In this way, calculating the optimal policy of actions distributions becomes an inference problem of calculating the posterior , which reads as, conditional on optimality from current time step until end of episode, and the current current state to be , the distribution of action , and this posterior corresponds to the optimal policy. Observing the d-separation from Figure 7, is conditionally independent of given , , so
IV-B Message passing for exact inference on the posterior
In this section, we give detailed derivation on doing exact inference on the policy posterior which is not given in . Although the results are not used due to unexpected behavior, there is theoretical insights that is worth being noted.
The graph in Figure 7 is similar to Hidden Markov Models (HMM) , if we could treat the tuple of variable as the latent variable counterpart of a HMM, with emission probability , while the transition probability, is from the variable tuple to a subcomponent of the ”latent” variable tuple .
Similar to the forward-backward message passing algorithm  in Hidden Markov Models , the posterior could also be calculated by passing messages. We offer a detailed derivation of the decomposition of the posterior in Equation (30), which is not available in .
In Hidden Markov Models (HMM) , if we use to represent the visible observed state and to represent the hidden latent state, for the series length, then it is essential to calculate the posterior and , which is the marginal of the complete posterior . The posterior marginal could be computed by the forward message and the backward message , which is the probability distribution of observables from current time step until the end of the sequence, conditional on the current latent state.
In contrast, here, only the backward messages are relevant. Additionally, the backward message here is not a probability distribution as in HMM, instead, is just a probability. In Figure 7, the backward message could be decomposed recursively. Since in  the author only give the conclusion without derivation, we give a detailed derivaion of this recursion in Equation (31).
The recursion in Equation (31) start from the last time point of an episode.
IV-C Connection between Message Passing and Bellman equation
then the corresponding policy could be written as Equation (34).
IV-D Variational approximation to ”optimal” policy
Since the exact inference lead to unexpected behavior, approximate inference could be used. The optimization of the policy could be considered as a variational inference problem, and we use the variational policy of the action posterior distribution , which could be represented by a neural network, to compose the proposal variational likelihood of the trajectory as in Equation (37):
, where the initial state distribution and the environmental dynamics of state transmission is kept intact. Using the proposal trajectory as a pivot, we could derive the Evidence Lower Bound (ELBO) of the optimal trajectory as in Equation (38), which correspond to an interesting objective function of reward plus entropy return, as in Equation (39).
Setting policy as Equation (34) lead to policy improvement. We offer a detailed proof for a key formula in Equation (42), which is stated in Equation (19) of  without proof. In Equation (42), we use to implicitly represent to avoid symbol aliasing whenever necessary.
V Variational Inference on the Environment
Another direction of using Variational Inference in Reinforcement Learning is to learn an environmental model, either on the dynamics or the latent state space posterior, instead of approximating the maximum entropy policy posterior in , explained in Section IV.
V-A Variational inference on transition model
In Variational Information Maximizing Exploration (VIME) , where dynamic model for the agent’s interaction with the environment is modeled using Bayesian Neural Network . The R.V. for is denoted by , and is treated in a Bayesian way by modeling the weight uncertainty. We represent this model with the graphical model in Figure 8, which is not given in . The belief about the environment is modeled as entropy of the neural network weights posterior distribution based on trajectory observations . The method encourages taking exploratory actions by alleviating the average information gain of the agent’s belief about the environment after observing a new state , which is , and this is equivalent to the entropy minus conditional entropy .
We derive in Equation (4344) that the entropy difference is actually the average information gain, which is equal to the mutual information between environmental parameter and the new state . Such a derivation is not given in .
Based on Equation (43), an intrinsic reward can be augmented from the environmental reward function, thus the method could be incorporated with any existing reinforcement learning algorithms for exploration, TRPO , for example. Upon additional observation of action and state pair on top of trajectory history , the posterior on the distribution of the environmental parameter , , could be updated to be in a Bayesian way as derived in Equation (45), which is first proposed in .
The last step of Equation (46) makes use of .
Since the integral in Equation (46) is not tractable, variational treatment over the neural network weights posterior distribution is used, characterized by variational parameter , as shown in the dotted line in Figure 8. The variational posterior about the model parameter , updated at each step, could than be used to calculate the intrinsic reward in Equation (43).
V-B Variational Inference on hidden state posterior
In Variational State Tabulation (VaST) , the author assume the high dimensional observed state to be represented by Observable , while the transition happens at the latent state space represented by , which is finite and discrete. The author assume a factorized form of observation and latent space joint probability, which we explicitly state in Equation (47).
Additionally, we characterize Equation (47) with the probabilistic graphical model in Figure 9 which does not exist in . Compared to Figure 7, here the latent state is in discrete space instead of high dimension, and the observation is a high dimensional image instead of binary variable to indicate optimal action. By assuming a factorized form of the variational posterior in Equation (48),
The author assume the episode length to be , and default frame prior observation to be blank frames. The Evidence Lower Bound (ELBO) of the observed trajectory of Equation (47) could be easily represented by a Varitional AutoEncoder  like architecture, where the encoder , together with the reparametrization trick , maps the observed state into parameters for the Con-crete distribution , so backprobagation could be used on deterministic variables to update the weight of the network based on the ELBO, which is decomposed into different parts of the reconstruction losses of the variational autoencoder like architecture. Like VIME , VaSt could be combined with other reinforcement learning algorithms. Here prioritized sweeping  is carried out on the Heviside activation of the encoder output directly, by counting the transition frequency, instead of waiting for the slowly learned environmental transition model in Equation (47). A potential problem of doing so is aliasing between latent state and observed state . To alleviate this problem, in , the author actively relabel the transition history in the replay memory once found the observable has been assigned a different latent discrete state.
As a tutorial survey, this paper introduces the application of Probabilistic Graphical Model and Variational Inference in Deep Reinforcement Learning. We reformulates some key concepts in Reinforcement Learning with Probabilistic Graphical Models, summarizes recent advances of Deep Reinforcement Learning and compares some representative methods from different aspects. We offer some detailed derivations and Probabilistic Graphical Models to those methods using variational inference, which are not included in the original contribution.
-  (2017) Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048–5058. Cited by: §II-E3, §II-E4.
-  (2017) A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866. Cited by: §I.
-  (2006) Pattern recognition and machine learning. springer. Cited by: 1st item, §I-B, §II-A2, §III, §III, §IV-A, §IV-B, §IV-B, §IV-B.
-  (2017) Variational inference: a review for statisticians. Journal of the American Statistical Association 112 (518), pp. 859–877. Cited by: 2nd item, §I-B.
-  (2015) Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424. Cited by: §I-B, §V-A.
-  (2018) Self-consistent trajectory autoencoder: hierarchical reinforcement learning with trajectory embeddings. arXiv preprint arXiv:1806.02813. Cited by: §II-A1.
-  (2018) Efficient model-based deep reinforcement learning with variational state tabulation. arXiv preprint arXiv:1802.04325. Cited by: 1st item, §V-B.
-  (2016) Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908. Cited by: §V-B.
-  (2017) Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1352–1361. Cited by: §IV-A, §IV-E, §IV-E, §IV-E.
-  (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290. Cited by: §IV-E.
-  (2016) Vime: variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109–1117. Cited by: 1st item, 2nd item, §V-A, §V-A, §V-B.
-  (1998) Planning and acting in partially observable stochastic domains. Artificial i ntelligence 101 (1-2), pp. 99–134. Cited by: §II-A2.
-  (2002) Approximately optimal approximate reinforcement learning. In ICML, Vol. 2, pp. 267–274. Cited by: §II-D, §II-D.
-  (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. Cited by: §I-B, §I.
-  (2018) Reinforcement learning and control as probabilistic inference: tutorial and review. arXiv preprint arXiv:1805.00909. Cited by: 2nd item, §II-A1, §IV-A, §IV-B, §IV-B, §IV-B, §IV-B, §IV-C, §IV-C, §V.
-  (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971. Cited by: §II-E2.
-  (2016) The concrete distribution: a continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712. Cited by: §V-B.
-  (2016) Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928–1937. Cited by: §II-E1.
-  (2015) Human-level control through deep reinforcement learning. Nature 518 (7540), pp. 529. Cited by: §I, §II-E2, §II-E4, §II-E.
-  (2004) Reinforcement learning with factored states and actions. Journal of Machine Learning Research 5 (Aug), pp. 1063–1088. Cited by: §III, §III, §IV-A.
-  (2015) Universal value function approximators. In International Conference on Machine Learning, pp. 1312–1320. Cited by: §II-E3.
-  (2015) Prioritized experience replay. arXiv preprint arXiv:1511.05952. Cited by: §II-E4.
-  (2015) Trust region policy optimization. In International conference on machine learning, pp. 1889–1897. Cited by: §I, §II-D, §II-E1, §II-E6, §V-A.
-  (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §II-E1.
-  (2014) Deterministic policy gradient algorithms. Cited by: §II-E2.
-  (2019) High dimensional restrictive federated model selection with multi-objective bayesian optimization over shifted distributions. arXiv preprint arXiv:1902.08999. Cited by: §II-E6.
-  (2019) Variational resampling based assessment of deep neural networks under distribution shift. Cited by: §I-B.
-  (2019) ReinBo: machine learning pipeline search and configuration with bayesian optimization embedded reinforcement learning. Cited by: §I.
-  (2011) Planning to be surprised: optimal bayesian exploration in dynamic environments. In International Conference on Artificial General Intelligence, pp. 41–51. Cited by: §V-A.
-  (1998) Introduction to reinforcement learning. Vol. 2, MIT press Cambridge. Cited by: 2nd item, §II-A1, §II-A2, §II-B, §II-B, §II-B, §II-C, §II-C, §III, §IV-E, §V-B.
-  (2000) Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, pp. 1057–1063. Cited by: §II-C.
-  (2016) Deep reinforcement learning with double q-learning. In Thirtieth AAAI conference on artificial intelligence, Cited by: §II-E.
-  (2019) Maximum entropy-regularized multi-goal reinforcement learning. arXiv preprint arXiv:1905.08786. Cited by: §II-A1, §II-E3, §II-E4.