Abstract
Deep reinforcement learning (DRL) has recently been adopted in a wide rangeof physics and engineering domains for its ability to solve decisionmakingproblems that were previously out of reach due to a combination ofnonlinearity and high dimensionality. In the last few years, it has spread inthe field of computational mechanics, and particularly in fluid dynamics, withrecent applications in flow control and shape optimization. In this work, weconduct a detailed review of existing DRL applications to fluid mechanicsproblems. In addition, we present recent results that further illustrate thepotential of DRL in Fluid Mechanics. The coupling methods used in each case arecovered, detailing their advantages and limitations. Our review also focuses onthe comparison with classical methods for optimal control and optimization.Finally, several test cases are described that illustrate recent progress madein this field. The goal of this publication is to provide an understanding ofDRL capabilities along with stateoftheart applications in fluid dynamics toresearchers wishing to address new problems with these methods.
Quick Read (beta)
A review on Deep Reinforcement Learning for Fluid Mechanics
Abstract
Deep reinforcement learning (DRL) has recently been adopted in a wide range of physics and engineering domains for its ability to solve decisionmaking problems that were previously out of reach due to a combination of nonlinearity and high dimensionality. In the last few years, it has spread in the field of computational mechanics, and particularly in fluid dynamics, with recent applications in flow control and shape optimization. In this work, we conduct a detailed review of existing DRL applications to fluid mechanics problems. In addition, we present recent results that further illustrate the potential of DRL in Fluid Mechanics. The coupling methods used in each case are covered, detailing their advantages and limitations. Our review also focuses on the comparison with classical methods for optimal control and optimization. Finally, several test cases are described that illustrate recent progress made in this field. The goal of this publication is to provide an understanding of DRL capabilities along with stateoftheart applications in fluid dynamics to researchers wishing to address new problems with these methods.
calc,fadings,shapes.misc,3d,arrows, decorations,decorations.pathmorphing, decorations.text,shapes.geometric,intersections, decorations.markings,patterns,spy,chains,positioning \usepgfplotslibraryfillbetween,polar,patchplots \setenumeratelabel=${\mathbf{\diamond}}$,itemsep=1pt,topsep=5pt
A review on Deep Reinforcement Learning for Fluid Mechanics
A Preprint
Paul Garnier
MINES Paristech , PSL  Research University
[email protected]
Jonathan Viquerat
MINES Paristech , PSL  Research University
CEMEF
[email protected]
Jean Rabault
Department of Mathematics, University of Oslo
and
MINES Paristech , PSL  Research University
CEMEF
[email protected]
Aurélien Larcher
MINES Paristech , PSL  Research University
CEMEF
[email protected]
Alexander Kuhnle
University of Cambridge
[email protected]
Elie Hachem^{†}^{†}thanks: Corresponding author
MINES Paristech , PSL  Research University
CEMEF
[email protected]
August 13, 2019
Keywords Deep Reinforcement Learning $\cdot $ Fluid Mechanics
1 Introduction
The process of learning and how the brain understands and classifies information has always been a source of fascination for humankind. Since the beginning of research in the field of artificial intelligence (AI), an algorithm able to learn to make decisions on its own has been one of the ultimate goals. Going back to the end of the 1980s (Sutton, 1988), reinforcement learning (RL) proposed a formal framework in which an agent learns by interacting with an environment through the gathering of experience (Françoislavet et al., 2018). Significant results were obtained during the following decade (Tesauro, 1995), although they were limited to lowdimensional problems.
In recent years, the rise of deep neural networks (DNN) has provided reinforcement learning with new powerful tools Goodfellow et al. (2017). The combination of deep learning with RL, called deep reinforcement learning (DRL), lifted several major obstacles that hindered classical RL by allowing the use of highdimensional state spaces and exploiting the feature extraction capabilities of DNNs. Recently, DRL managed to perform tasks with unprecedented efficiency in many domains such as robotics Pinto et al. (2017) or language processing Bahdanau et al. (2016), and even achieved superhuman levels in multiple games: Atari games Mnih et al. (2013), Go Silver et al. (2017), Dota II OpenAI (2018), Starcraft II Vinyals et al. (2019) and even Poker Brown & Sandholm (2019). Also in industrial application, DRL has proven itself useful. For example, Wayve trains autonomous cars both onboard (Kendall et al., 2018) and through simulation (Bewley et al., 2018), and Google uses DRL in order to control the cooling of its data centers (Knight, 2018).
In the field of fluid mechanics and mechanical engineering, one is also confronted with nonlinear problems of high dimensionality. For example, using computational simulations to test several different designs or configurations has proven a useful technique. However, the number of possibilities to explore can make such search difficult since it is often unfeasible to evaluate all configurations exhaustively. Therefore, the assistance of automatic optimization procedures is needed to help find optimal designs.
Possible applications range from multiphase flows in microfluidics to shape optimization in aerodynamics or conjugate heat transfer in heat exchangers, and many other fields can gain from such techniques such as biomechanics, energy, or marine technologies. This is the primary motivation for the combination of computational fluid dynamics (CFD) with numerical optimization methods. Most of the time, the main obstacle is the high computational cost of determining the sensitivity of the objective function to variations of the design parameters by repeated calculations of the flow. Besides, classical optimization techniques, like direct gradientbased methods, are known for their lack of robustness and for their tendency to fall into local optima. Generic and robust search methods, such as evolutionary or genetic algorithms, offer several attractive features and have been used widely for design shape optimization (Ali & Behdinan, 2003; Lee & Hajela, 1996; Mäkinen et al., 1999). In particular, they can be used for multiobjective multiparameter problems. They have been successfully tested for many practical cases, for example for design shape optimization in the aerospace (Matos et al., 2004) and automotive industries (Muyl et al., 2004). However, the use of a fully automatic evolutionary algorithms coupled with CFD for the optimization of multiobjective problems remains limited by the computing burden at stake and is still far from providing a practical tool for many engineering applications.
Despite great potential, the literature applying DRL to computational or experimental fluid dynamics remains limited. In recent years, only a handful of applications were proposed that take advantage of such learning methods. These applications include flow control problems (Rabault et al., 2019; Guéniat et al., 2016), process optimization (Novati et al., 2017; Lee et al., 2018), shape optimization (of Aeronautics & Astronautics, 2008), and our own results about laminar flows past a square cylinder presented later in this work, among others^{1}^{1} 1 These references and more are addressed in details in the remaining of this article..
This paper presents an introduction to modern RL for the neophyte reader to get a grasp of the main concepts and methods existing in the domain, including the necessary basics of deep learning. Subsequently, the relevant fluid mechanics concepts are reiterated. We then go through a thorough review of the literature and describe the content of each contribution. In particular, we cover the choice of DRL algorithm, the problem complexity, and the concept of reward shaping. Finally, an outline is given on the future possibilities of DRL coupled with fluid dynamics.
2 Deep Reinforcement Learning
In this section, the basic concepts of DRL are introduced. For the sake of brevity, some topics are only minimally covered, and the reader is referred to more detailed introductions to (D)RL (Sutton & Barto, 2018; Françoislavet et al., 2018). First, RL is presented in its mathematical description as a Markov Decision Process. Then, its combination with deep learning is detailed, and an overview of several DRL algorithms is presented.
$\gamma $  discount factor 

$\lambda $  learning rate 
$\alpha $, $\beta $  stepsize 
$\u03f5$  probability of random action 
$s$,${s}^{\prime}$  states 
${\mathcal{S}}^{+}$  set of all states 
$\mathcal{S}$  set of nontermination states 
$a$  action 
$\mathcal{A}$  set of all actions 
$r$  reward 
$\mathcal{R}$  set of all rewards 
$t$  time station 
$T$  final time station 
${a}_{t}$  action at time $t$ 
${s}_{t}$  state at time $t$ 
${r}_{t}$  reward at time $t$ 
$R(\tau )$  discounted cumulative reward following trajectory $\tau $ 
$R(s,a)$  reward received for taking action $a$ in state $s$ 
$\pi $  policy 
$\theta $, ${\theta}^{\prime}$  parameterization vector of a policy 
${\pi}_{\theta}$  policy parameterized by $\theta $ 
$\pi (s)$  action probability distribution in state $s$ following $\pi $ 
$\pi (as)$  probability of taking action $a$ in state $s$ following $\pi $ 
${V}^{\pi}(s)$  value of state $s$ under policy $\pi $ 
${V}^{*}(s)$  value of state $s$ under the optimal policy 
${Q}^{\pi}(s,a)$  value of taking action $a$ in state $s$ under policy $\pi $ 
${Q}^{*}(s,a)$  value of taking action $a$ in state $s$ under the optimal policy 
${Q}_{\theta}(s,a)$  estimated value of taking action $a$ in state $s$ with parameterization $\theta $ 
2.1 Reinforcement Learning
At its core, reinforcement learning models an agent interacting with an environment and receiving observations and rewards from it, as shown in figure 1. The goal of the agent is to determine the best action in any given state in order to maximize its cumulative reward over an episode, i.e. over one instance of the scenario in which the agent takes actions. Taking the illustrative example of a board game:

1.
An episode is equivalent to one game;

2.
A state consists of the summary of all pieces and their positions;

3.
Possible actions correspond to moving a piece of the game;

4.
The reward may simply be $+1$ if the agent wins, $1$ if it loses and $0$ in case of a tie or a nonterminal game state.
Generally, the reward is a signal of how ‘good’ a board situation is, which helps the agent to learn distinguish more promising from less attractive decision trajectories. A trajectory is a sequence of states and actions experienced by the agent:
$$\tau =({s}_{0},{a}_{0},{s}_{1},{a}_{1},\mathrm{\dots}).$$ 
The cumulative reward (i.e. the quantity to maximize) is expressed along a trajectory, and includes a discount factor $\gamma \in [0,1]$ that smoothes the impact of temporally distant rewards (it is then called discounted cumulative reward):
$$R(\tau )=\sum _{t=0}^{T}{\gamma}^{t}{r}_{t}.$$ 
RL methods are classically divided in two categories, namely modelfree and modelbased algorithms. Modelbased method incorporates a model of the environment they interact with, and will not be considered in this paper. On the contrary, modelfree algorithms directly interact with their environment, and are currently the most commonly used within the DRL community, mainly for their ease of application and implementation. Modelfree methods are further distinguished between valuebased methods and policybased methods (Sutton & Barto, 1998). Although both approaches aim at maximizing their expected return, policybased methods do so by directly optimizing the decision policy, while valuebased methods learn to estimate the expected value of a stateaction pair optimally, which in turn determines the best action to take in each state. Both families inherit from the formulation of Markov decision processes (MDP), which is detailed in the following section.
2.1.1 Markov decision processes
A MDP can be defined by a tuple $(\mathcal{S},\mathcal{A},\mathbb{P},R)$ (Bellman, 1957; Howard, 1960), where:

1.
$\mathcal{S}$ , $\mathcal{A}$ and $R$ are defined in table 1,

2.
$\mathbb{P}:\mathcal{S}\times \mathcal{A}\times {\mathcal{S}}^{+}\to [0,1]$, where $\mathbb{P}({s}^{\prime}s,a)$ is the probability of getting to state ${s}^{\prime}$ from state $s$ following action $a$.
In the formalism of MDP, the goal is to find a policy $\pi (s)$ that maximizes the expected reward. However, in the context of RL, $\mathbb{P}$ and $R$ are unknown to the agent, which is expected to come up with an efficient decisional process by interacting with the environment. The way the agent induces this decisional process classifies it either in valuebased or policybased methods.
2.1.2 Valuebased methods
In valuebased methods, the agent learns to optimally estimate a value function, which in turn dictates the policy of the agent by selecting the action of the highest value. One usually defines the state value function:
$${V}^{\pi}(s)=\underset{\tau \sim \pi}{\mathbb{E}}\left[R(\tau )s\right],$$ 
and the stateaction value function:
$${Q}^{\pi}(s,a)=\underset{\tau \sim \pi}{\mathbb{E}}\left[R(\tau )s,a\right],$$ 
which respectively denote the expected discounted cumulative reward starting in state $s$ (resp. starting in state $s$ and taking action $a$) and then follow trajectory $\tau $ according to policy $\pi $. It is fairly straightforward to see that these two concepts are linked as follows:
$${V}^{\pi}(s)=\underset{a\sim \pi}{\mathbb{E}}\left[{Q}^{\pi}(s,a)\right],$$ 
meaning that in practice, ${V}^{\pi}(s)$ is the weighted average of ${Q}^{\pi}(s,a)$ over all possible actions by the probability of each action. Finally, the stateaction advantage function can be defined as ${A}^{\pi}(s,a)={Q}^{\pi}(s,a){V}^{\pi}(s)$. One of the main valuebased methods in use is called Qlearning, as it relies on the learning of the Qfunction to find an optimal policy. In classical Qlearning, the Qfunction is stored in a Qtable, which is a simple array representing the estimated value of the optimal Qfunction ${Q}^{*}(s,a)$ for each pair $(s,a)\in \mathcal{S}\times \mathcal{A}$. The Qtable is initialized randomly, and its values are progressively updated as the agent explores the environment, until the Bellman optimality condition (Bellman & Dreyfus, 1962) is reached:
$${Q}^{*}(s,a)=R(s,a)+\gamma \underset{{a}^{\prime}}{\mathrm{max}}{Q}^{*}({s}^{\prime},{a}^{\prime}).$$  (1) 
The Bellman equation indicates that the Qtable estimate of the Qvalue has converged and that systematically taking the action with the highest Qvalue leads to the optimal policy. In practice, the expression of the Bellman equation (Bellman & Dreyfus, 1962) is used to update the Qtable estimates.
2.1.3 Policybased methods
Another approach consists of directly optimizing a parameterized policy ${\pi}_{\theta}(as)$, instead of resorting to a value function estimate to determine the optimal policy. In contrast with valuebased methods, policybased methods offer three main advantages:

1.
They have better convergence properties, although they tend to be trapped in local minima;

2.
They naturally handle high dimensional action spaces;

3.
They can learn stochastic policies.
To determine how "good" a policy is, one defines an objective function based on the expected cumulative reward:
$$J(\theta )=\underset{\tau \sim {\pi}_{\theta}}{\mathbb{E}}\left[R(\tau )\right],$$ 
and seeks the optimal parameterization ${\theta}^{*}$ that maximizes $J(\theta )$:
$${\theta}^{*}=\mathrm{arg}\underset{\theta}{\mathrm{max}}\mathbb{E}\left[R(\tau )\right].$$ 
To that end, the gradient of the cost function is needed. This is not an obvious task at first, as one is looking for the gradient with respect to the policy parameters $\theta $, in a context where the effects of policy changes on the state distribution are unknown. Indeed, modifying the policy will most certainly modify the set of visited states, which could affect performance in an unknown manner. This derivation is a classical exercise that relies on the logprobability trick, which allows expressing ${\nabla}_{\theta}J(\theta )$ as an evaluable expected value:
$${\nabla}_{\theta}J(\theta )=\underset{\tau \sim {\pi}_{\theta}}{\mathbb{E}}\left[\sum _{t=0}^{T}{\nabla}_{\theta}\mathrm{log}\left({\pi}_{\theta}({a}_{t}{s}_{t})\right)R(\tau )\right].$$  (2) 
This gradient is then used to update the policy parameters:
$$\theta \leftarrow \theta +\lambda {\nabla}_{\theta}J(\theta ).$$  (3) 
As ${\nabla}_{\theta}J(\theta )$ takes the form of expected value, in practice, it is evaluated by averaging its argument over a set of trajectories. In such vanilla policy gradient methods, the actions are evaluated at the end of the episode (they belong to MonteCarlo methods). If some lowquality actions are taken along the trajectory; their negative impact will be averaged by the highquality actions and will remain undetected. This problem is overcome by actorcritic methods, in which a Qfunction evaluation is used in conjunction with a policy optimization.
2.2 Deep Reinforcement Learning
In RL, the evaluation of value functions such as ${V}^{\pi}$ or ${Q}^{\pi}$ can be done in several ways. One possible way, as presented earlier, is to use tables that store the values for every state or actionstate pair. However, this strategy does not scale with the sizes of state and action spaces. Another possibility is to use neural networks to estimate value functions, or, for policybased methods, to output an action distribution given input states. Artificial neural networks can be seen as universal function approximators Siegelmann & Sontag (1995), which means that, if properly trained, they can represent arbitrarily complex mappings between spaces. This particularly suitable property of ANNs gave rise to deep reinforcement learning, i.e. reinforcement learning algorithms using deep neural networks as functions approximators. Regarding the notions introduced in previous sections, it is natural to think of $\theta $ as the parameterization of the neural network, i.e. as it set of weights and biases. In the following section, a basic introduction to neural networks is given. The reader is referred to Goodfellow et al. (2017) for additional details.
2.2.1 Artificial neurons and neural networks
The basic unit of a neural network (NN) is the neuron, which representation is given in figure 2. An input vector $x$, associated with a set of weights $w$, is provided to the neuron. The neuron then computes the weighted sum $w\cdot x+b$, where $b$ is called the bias and applies the activation function $\sigma $ to this sum. This is the output of the neuron, hereafter noted $z$. In the neuron, the weights and the bias represent the degrees of freedom (i.e. , the parameters that can be adjusted to approximate the function $f$), while the activation function is a hyperparameter, i.e. , it is part of the choices made during the network design.
In their purest form, neural networks consist of several layers of neurons connected to each other, as shown in the primary example of 2. Such a network is said to be fully connected (FC), in the sense that each neuron of a layer is connected to all the neurons of the following layer. As it was said in the last section, each connection is characterized by a weight, and in addition each neuron has a bias, except for the input layer. Indeed, as a convention, the input layer is usually drawn as a regular layer, but it does not hold biases, nor is an activation function applied at its output. The learning process in neural networks consists in adjusting all the biases and weights of the network in order to reduce the value of a wellchosen loss function that represents the quality of the network prediction. This update is usually performed by a stochastic gradient method, in which the gradients of the loss function with respect to the weights and biases are estimated using a backpropagation algorithm.
When working for instance with images as input, it is customary to exploit convolutional layers instead of FC ones. Rather than looking for patterns in their entire input space as FC layers, convolutional ones can extract local features. Additionally, they can build a hierarchy of increasingly sophisticated features. In such layers, a convolution kernel (i.e. , a tensor product with a weight matrix) is applied on a small patch of the input image and is used as input for a neuron of the next layer. The patch is then moved, and the operation repeated until the input image has been entirely covered. The size of the patch is usually known as the kernel size. The complete coverage of the image with this process generates a kernel, also called filter or feature map. In most cases, multiple kernels are generated at each layer, each encoding a specific feature of the input image. When used for regression applications, convolutional networks (or convnets) most often end with a fully connected layer, followed by the output layer, which size is determined by that of the sought quantity of interest. These considerations are discussed in details in a large variety of books and articles such as Goodfellow et al. (2017).
2.3 DRL algorithms
In this section, the main lines of the major DRL algorithms in use today are presented. For the sake of brevity, many details and discussions are absent from the following discussions. The reader is referred to the profuse literature on the topic.
2.3.1 Deep Qnetworks
Instead of updating a table holding Qvalues for each possible $(s,a)$ pair, ANNs can be used to generate a map ${\mathcal{S}}^{+}\times \mathcal{A}\u27f6\mathbb{R}$ (called deep Qnetworks (DQN)) that provides an estimate of the Qvalue for each possible action given an input state. As it is usual for neural networks, the update of the Qnetwork requires a loss function for the gradient descent algorithm, that will be used to optimize the network parameters $\theta $. To do so, it is usual to exploit the Bellman optimality equation (1):
$$L(\theta )=\mathbb{E}\left[\frac{1}{2}{\left(\underset{\text{Target}}{\underset{\u23df}{\left[R(s,a)+\gamma \underset{{a}^{\prime}}{\mathrm{max}}{Q}_{\theta}({s}^{\prime},{a}^{\prime})\right]}}{Q}_{\theta}(s,a)\right)}^{2}\right].$$  (4) 
In the latter expression, ${Q}_{\theta}(s,a)$ represents the Qvalue estimate provided by the DQN for action $s$ and state $a$ under network parameterization $\theta $. It must be noted that the quantity denoted target is the same that appears in the Bellman optimality equation (1): when the optimal set of parameters ${\theta}^{*}$ is reached, ${Q}_{\theta}(s,a)$ is equal to the target, and $L(\theta )$ is equal to zero.
In vanilla deep Qlearning (and more generally in Qlearning), an exploration/exploitation tradeoff must be implemented, to ensure a sufficient exploration of the environment. To do so, a parameter $\u03f5\in [0,1]$ is defined, and a random value is drawn in the same range before each action. If this value is below $\u03f5$, a random action is taken. Otherwise, the algorithm follows the action prescribed by ${\mathrm{max}}_{a}{Q}_{\theta}(s,a)$. Most often, $\u03f5$ follows a schedule, starting with high values at the beginning of learning, and progressively decreasing. The vanilla algorithm using DQN is shown below :
{algorithmic}[1] \StateInitialize actionvalue function parameters ${\theta}_{0}$ \For$k$ = $0$, $M$ \CommentLoop over episodes \For$t$ = $0$, $T$ \CommentLoop over time stations \StateDraw random value $\omega \in [0,1]$ \Comment$\u03f5$greedy strategy \If($$) \StateChoose action ${a}_{t}$ randomly \Else\StateChoose action ${a}_{t}={\mathrm{max}}_{a}{Q}_{\theta}({s}_{t},a)$ \EndIf\StateExecute action ${a}_{t}$ and observe reward ${r}_{t}$ and state ${s}_{t+1}$ \StateForm target ${y}_{t}={r}_{t}+{\mathrm{max}}_{a}{Q}_{\theta}({s}_{t+1},a)$ \StatePerform gradient descent using ${({y}_{t}{Q}_{\theta}({s}_{t},{a}_{t}))}^{2}$ \StateUpdate $\theta $ and ${s}_{t}$ \EndFor\EndFor
Still, the basic deep Qlearning algorithm presents several flaws, and many improvements can be added to it:

1.
Experience replay: To avoid forgetting about its past learning, a replay buffer can be created that contains random previous experiences. This buffer is regularly fed to the network as learning material, so previously acquired behaviors are not erased by new ones. This improvement also reduces the correlation between experiences: as the replay buffer is randomly shuffled, the network experiences past $({s}_{t},{a}_{t},{r}_{t},{s}_{t+1})$ tuples in a different order, and is therefore less prone to learn correlation between these (Tsitsiklis & Van Roy, 1997; Lin, 1993);

2.
Fixed Qtargets: When computing the loss (4), the same network is used to evaluate the target and ${Q}_{\theta}({s}_{t},{a}_{t})$. Hence, at every learning step, both the Qvalue and the target move, i.e. the gradient descent algorithm is chasing a moving target, implying big oscillations in the training. To overcome this, a separate network is used to evaluate the target (target network), and its weight parameters ${\theta}^{}$ are updated less frequently than that of the Qnetwork. This way, the target remains fixed for several Qnetwork updates, making the learning easier;

3.
Double DQN: The risk of using a single Qnetwork is that it may overestimate the Qvalues, leading to suboptimal policies, as ${\pi}_{\theta}$ is derived by systematically choosing the actions of highest Qvalues. In (van Hasselt et al., 2015), two networks are used jointly: a DQN is used to select the best action, while the other one is used to estimate the target value;

4.
Prioritized replay buffer: In the exploration of the environment by the agent, some experiences might occur rarely while being of high importance. As the memory replay batch is sampled uniformly, these experiences may never be selected. In Schaul et al. (2015), the authors propose to sample the replay buffer according to the training loss value, in order to fill the buffer in priority with experiences where the Qvalue evaluation was poor, meaning that the network will have better training on "unexpected events". To avoid overfitting on these events, the filling of the prioritized buffer follows a stochastic rule.
2.3.2 Deep policy gradient
When the policy ${\pi}_{\theta}$ is represented by a deep neural network with parameters $\theta $, the evaluation of ${\nabla}_{\theta}J(\theta )$ can be delegated to the backpropagation algorithm (see section 2.2.1), as long as the gradient of the loss function equals the policy gradient (2). The loss can be expressed as:
$$L(\theta )=\underset{\tau \sim {\pi}_{\theta}}{\mathbb{E}}\left[\sum _{t=0}^{T}\mathrm{log}\left({\pi}_{\theta}({a}_{t}{s}_{t})\right)R(\tau )\right].$$ 
However, it can be shown that using the full history of discounted reward along trajectory $\tau $ is not necessary, as a given action ${a}_{t}$ has no influence on the rewards obtained at previous timestations. Several expressions are possible to use in the stead of $R(\tau )$, that do not modify the value of the computed policy gradient, while reducing its variance (and therefore the required amount of trajectories to correctly approximate the expected value) (Schulman et al., 2015). Among those, the advantage function $A(s,a)$ is commonly chosen for its low variance:
$$A(s,a)=Q(s,a)V(s).$$ 
The advantage function $A(s,a)$, represents the improvement obtained in the expected cumulative reward when taking action $a$ in state $s$, compared to the average of all possible actions taken in state $s$. In practice, it is not readily available and must be estimated using the value function. In practice, this estimate is provided by a separately learned value function. As a result, the loss function writes as:
$$L(\theta )=\underset{\tau \sim {\pi}_{\theta}}{\mathbb{E}}\left[\sum _{t=0}^{T}\mathrm{log}\left({\pi}_{\theta}({a}_{t}{s}_{t})\right){A}^{{\pi}_{\theta}}({s}_{t},{a}_{t})\right].$$ 
The vanilla deep policy gradient algorithm is presented below:
{algorithmic}[1] \StateInitialize policy parameters ${\theta}_{0}$ \For$k$ = $0$, $M$ \CommentLoop over batches of trajectories \For$i$ = $0$, $m$ \CommentLoop over trajectories in batch $k$ \StateExecute policy ${\pi}_{{\theta}_{k}}$ \StateCollect advantage estimates ${A}^{{\pi}_{{\theta}_{k}}}({s}_{t},{a}_{t})$ \EndFor\StateEstimate $L({\theta}_{k})$ over current batch of trajectories \StateUpdate policy parameters ${\theta}_{k}$ using stochastic gradient ascent \EndFor
2.3.3 Advantage actorcritic
As mentioned in section 2.1.3, actorcritic methods propose to simultaneously exploit two networks to improve the learning performance. One of them is policybased, while the other one is valuebased:

1.
The actor ${\pi}_{\theta}(s,a)$, controls the actions taken by the agent;

2.
The critic ${Q}_{w}(s,a)$, evaluates the quality of the action taken by the actor.
Both these networks are updated in parallel, in a timedifference (TD) fashion (one update at each time station) instead of the usual MonteCarlo configuration (one update at the end of the episode). In practice, the advantage function $A(s,a)$ is preferred to the Qvalue for its lower variability. In order to avoid having to evaluate two value functions, the advantage function is usually approximated, using the reward obtained by the actor after action ${a}_{t}$ and the value evaluated by the critic in state ${s}_{t+1}$:
$${A}_{w}({s}_{t},{a}_{t})\sim R({s}_{t},{a}_{t})+\gamma {V}_{w}({s}_{t+1}){V}_{w}({s}_{t}).$$ 
The actorcritic process then goes as follows:
{algorithmic}[1] \StateInitialize policy parameters ${\theta}_{0}$ \CommentActor \StateInitialize actionvalue function parameters ${w}_{0}$ \CommentCritic \For$k$ = $0$, $M$ \CommentLoop over episodes \For$t$ = $0$, $T$ \CommentLoop over time stations \StateExecute action ${a}_{t}$ following ${\pi}_{\theta}$, get reward ${r}_{t}$ and state ${s}_{t+1}$ \CommentActor \StateEvaluate target $R({s}_{t},{a}_{t})+\gamma {V}_{w}({s}_{t+1})$ \CommentCritic \StateUpdate $w$ to ${w}^{\prime}$ using target \CommentCritic \StateEvaluate advantage ${A}_{{w}^{\prime}}({s}_{t},{a}_{t})=R({s}_{t},{a}_{t})+\gamma {V}_{{w}^{\prime}}({s}_{t+1}){V}_{{w}^{\prime}}({s}_{t})$ \CommentCritic \StateUpdate $\theta $ to ${\theta}^{\prime}$ using ${A}_{{w}^{\prime}}({s}_{t},{a}_{t})$ \CommentActor \EndFor\EndFor
In the latter algorithm, the actor and critic parameter updates follow the methods described in sections 2.3.1 and 2.3.2. The advantage actor critic algorithm has been developed in asynchronous (A3C Mnih et al. (2016)) and synchronous (A2C) versions. The version of the algorithm presented here corresponds to a timedifference A2C, although many possible variants exist, such as the nstep A2C.
2.3.4 Trustregion and proximal policy optimization
Trust region policy optimization (TRPO) was introduced in 2015 as a major improvement of vanilla policy gradient Schulman et al. (2015). In this method, the network update exploits a surrogate advantage functional:
$${\theta}_{k+1}=\mathrm{arg}\underset{\theta}{\mathrm{max}}L({\theta}_{k},\theta ),$$ 
with
$$L({\theta}_{k},\theta )=\underset{(s,a)\sim {\pi}_{{\theta}_{k}}}{\mathbb{E}}\left[\mathrm{\Pi}(s,a,\theta ,{\theta}_{k}){A}^{{\pi}_{{\theta}_{k}}}(s,a)\right],$$ 
and
$$\mathrm{\Pi}(s,a,\theta ,{\theta}_{k})=\frac{{\pi}_{\theta}(as)}{{\pi}_{{\theta}_{k}}(as)}.$$ 
In latter expression, $L({\theta}_{k},\theta )$ measures how much better (or worse) the policy ${\pi}_{\theta}$ performs compared to the previous policy ${\pi}_{{\theta}_{k}}$. In order to avoid too large policy updates that could collapse the policy performance, TRPO leverages secondorder natural gradient optimization to update parameters within a trustregion of a fixed maximum KullbackLeibler divergence between old and updated policy distribution. This relatively complex approach was replaced in the PPO method by simply clipping the maximized expression:
$$L({\theta}_{k},\theta )=\underset{(s,a)\sim {\pi}_{{\theta}_{k}}}{\mathbb{E}}\left[\mathrm{min}(\mathrm{\Pi}(s,a,\theta ,{\theta}_{k}){A}^{{\pi}_{{\theta}_{k}}}(s,a),g(\epsilon ,{A}^{{\pi}_{{\theta}_{k}}}(s,a)))\right],$$ 
where
$$ 
where $\epsilon $ is a small, userdefined parameter. When ${A}^{{\pi}_{{\theta}_{k}}}(s,a)$ is positive, taking action $a$ in state $s$ is preferable to the average of all actions that could be taken in that state, and it is natural to update the policy to favor this action. Still, if the ratio $\mathrm{\Pi}(s,a,\theta ,{\theta}_{k})$ is very large, stepping too far from the previous policy ${\pi}_{{\theta}_{k}}$ could damage performance. For that reason, $\mathrm{\Pi}(s,a,\theta ,{\theta}_{k})$ is clipped to $1+\epsilon $ to avoid too large updates of the policy. If ${A}^{{\pi}_{{\theta}_{k}}}(s,a)$ is negative, taking action $a$ in state $s$ represents a poorer choice than the average of all actions that could be taken in that state, and it is natural to update the policy to decrease the probability of taking this action. In the same fashion, $\mathrm{\Pi}(s,a,\theta ,{\theta}_{k})$ is clipped to $1\epsilon $ if it happens to be lower than that value.
Many refinements can be added to these methods that are out of reach of this introduction, such as the use of generalized advantage estimator (Schulman et al., 2016), the introduction of evolution strategies in the learning process (Houthooft et al., 2018), or the combination of DQN with PG methods for continuous action spaces (Lillicrap et al., 2015).
2.4 A conclusion on DRL
Since the first edition of the book of Sutton Sutton & Barto (1998), RL has become a particularly active field of research. However, the domain really started striving after the paper from Mnih et al. (2013) and its groundbreaking results using DQN to play Atari games. As of today, the domain holds countless major achievements, and the ability of DRL to learn to achieve complex tasks is well established. It is now exploited in multiple fields of research, and is featured in an everincreasing amount of publications, as shown in figure 3.
A major advantage of DRL is that it can be used as an agnostic tool for control and optimization tasks, both in continuous and discrete contexts. Its only requirement is a welldefined $({s}_{t},{a}_{t},{r}_{t})$ interface from the environment, which can consist in a numerical simulation or a reallife experiment. As seen earlier, both DQN and DPG ensure a good exploration of the states space, making them able to unveil new optimal behaviors. Moreover, DRL is also robust to transfer learning, meaning that an agent trained on one problem will train much faster on a similar environment. This represents a great strength, especially for domains like fluid mechanics, where computational time represents a limiting factor (section 3.7 illustrates this point). Transfer learning also introduces a large contrast between DRL and regular optimization techniques, such as adjoint methods, where the product of optimization in a given case cannot be reused to speed up optimization in a similar configuration. Finally, as illustrated in section 3.6, DRL libraries can exploit parallel learning with a closetoperfect scaling on the available resources. Hence, for its ease of use and its numerous capabilities, DRL represents a promising tool for optimization and design processes involving computational or experimental environments.
3 Applications
In this section, several applications combining DRL and fluid mechanics found in the literature are presented in details. For each case, the numerical experiments and the obtained results are detailed. The choice of DRL algorithm and the problem complexity are also considered. Table 2 presents an overview of the reviewed articles and their corresponding references.
Main topic  Algorithm  References 

Flow Control  QL  Guéniat et al. (2016) 
DQN  Gazzola et al. (2016), Novati et al. (2017)  
ADQN  Verma et al. (2018)  
TRPO  Ma et al. (2018)  
A3C  Garnier & Viquerat (2019)  
PPO  Rabault et al. (2019), Rabault & Kuhnle (2019)  
Homogeneity optimization  QL  Colabrese et al. (2017), Gustavsson et al. (2017) 
QL  Qiu et al. (2018), Hou Tsang et al. (2018)  
Shape Optimization  QL  of Aeronautics & Astronautics (2008) 
Chaotic Systems  DDPG  Bucci et al. (2019) 
3.1 Synchronised swimming of two fish  Novati et al. (2017)
In Novati et al. (2017), the authors study the swimming kinematics of two fish in a viscous incompressible flow. The first fish is the leader, and is affected a prescribed gait, while the second fish is a follower, which dynamics are unknown. An objective of the paper is to exploit DRL to derive a swimming strategy that reduces the energy expenditure of the follower. A twodimensional NavierStokes equation is solved to simulate the viscous incompressible flow using a remeshed vortex method. To represent the fish and its undulation, a simplified physical model is used that describes the body curvature $k(s,t)$ of the fish:
$$k(s,t)=A(s)\mathrm{sin}\left[2\pi \left(\frac{t}{{T}_{p}}\frac{s}{L}\right)+\varphi \right],$$  (5) 
where $L$ is the length of the fish, ${T}_{p}$ is the tailbeat frequency, $\varphi $ is a phasedifference, $t$ is the time variable, and $s$ is the abscissa. The observation of the environment provided to the DRL agent is composed of:

1.
The displacements $\mathrm{\Delta}x$ and $\mathrm{\Delta}y$ (cf figure 4),

2.
The orientation $\theta $ between the follower and the leader (cf figure 4),

3.
The moment where the action is going to take place (during one period of a tailbeat undulation),

4.
The last two actions taken by the agent.
The agent provides actions that correspond to an additive curvature term for expression (5):
$${k}^{\prime}(s,t)=A(s)M(\frac{t}{{T}_{p}}\frac{s}{L}).$$ 
The reward to maximize is defined as ${r}_{t}=12\frac{\mathrm{\Delta}y}{L}$, which penalizes the follower when it laterally strays away from the path of the leader. Finally, the state space is artificially restricted by terminating the current episode with an arbitrary reward ${r}_{t}=1$ when the two fish get too far away from each other. In this application, a DQN algorithm is used to control the follower. For the best strategy found by the agent, swimming efficiency was increased by 20% compared to baseline of multiple solitary swimmers.
3.2 Efficient collective swimming by harnessing vortices through deep reinforcement learning Verma et al. (2018)
The author in Verma et al. (2018) further investigated the swimming strategies of multiple fishes. They work with two and threedimensional flows (see figure 5), and analyze with many details the movement of trained fishes. The DRL model is identical to what is used in Novati et al. (2017), but the neural network used was improved by using recurrent neural networks with LongShort Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997). The authors underline the importance of this feature, arguing that here, past observations hold informations that are relevant for future transitions (i.e. the process is not Markovian anymore). Two different rewards are used: the first one is identical to that of last section, while the second one is simply equal to the swimming efficiency of the follower (see Verma et al. (2018)). The result obtained show that collective energysavings could be achieved with an appropriate use of the wake generated by other swimmers, when compared to the case of multiple solitary swimmers.
3.3 Fluid directed rigid body control using deep reinforcement learning Ma et al. (2018)
The control of complex environments that include two fluid flows interacting with multiple rigid bodies (cf figure 6) was first proposed by Ma et al. (2018). In this contribution, a multiphase NavierStokes solver is used to simulate the full characteristics of the interactions between fluids and rigid bodies at different scales. To provide a rich set of observations to the agent while limiting its input dimension, a convolutional autoencoder is used to extract lowdimensional features from the highdimensional velocity field of the fluid (Liou et al., 2014). This technique allows the use of a smaller neural network with faster convergence. The autoencoder is trained using a TRPO agent, which received two different observations: (i) extracted features from the fluid velocity field, and (ii) features from the rigid body.
In the environment, the agent is given control over several fluid jets (see figure 6), each of them described by 3 parameters: (i) the jet lateral acceleration ${\ddot{x}}_{jet}$, (ii) the jet angular acceleration ${\ddot{\beta}}_{jet}$, and (iii) the use of the jet ${\delta}_{jet}$. Using lateral and angular accelerations instead of lateral and angular positions allows the jet to move smoothly even with a noisy policy. Different rewards were designed depending on the environment goals. A possible example consists in having a rigid body balanced at a fixed position. In this case the designed reward is:
$${r}_{t}={w}_{c}\mathrm{exp}({c{c}^{*}}^{2})+{w}_{v}\mathrm{exp}({v}^{2})+{w}_{e}(1{\delta}_{jet}),$$  (6) 
where the first term penalizes a position mismatch according to the goal ${c}^{*}$, the second penalizes the presence of a residual velocity, and the last one encourages the agent to use the smallest amount of control forces. ${w}_{c}$, ${w}_{v}$ and ${w}_{e}$ are userdefined parameters used to weight the three contributions.
Overall, impressive results were obtained on complex experiments, including playing music with the system or keeping a rotating object at a given position. An important feature of this contribution is the use of the autoencoder to extract features from the highdimensional velocity fluid fields. Of particular interest is the simultaneous training of the two networks (autoencoder and policy network), leading to a high convergence rate of the overall method.
3.4 Flow shape design for microfluidic devices using deep reinforcement learning Lee et al. (2018)
In Lee et al. (2018), the authors explore the capabilities of DRL for microfluidic flow sculpting using pillarshaped obstacles to deform an input flow. Given a target flow, a double DQN is used to generate the corresponding optimal sequence of pillars. The observations provided to the agent consist in the current flow shape, represented as a matrix of black and white pixels. In return, the actions correspond to pillar configurations: the agent can add up to 32 pillars in order to reshape the input flow. A pixel match rate (PMR) function is used to evaluate the similarity between the target flow and the current flow. The reward is then defined using that PMR as:
$${r}_{t}=(1\frac{\text{PMR}b}{b}),$$ 
where $b$ is the PMR of the worstcase scenario performed. PMR over 90% were obtained for different target flows, showing that DRL represents a viable alternative to other optimization processes such as genetic algorithms (see figure 7). Transfer learning is also exploited, showing that an agent trained on a first flow could be retrained much more efficiently on a different flow than an untrained one, thus saving a lot of computational time.
3.5 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control (Rabault et al., 2019)
In Rabault et al. (2019), the authors present the first application of DRL for performing active flow control in a simple CFD simulation. For this, a simple benchmark is used. More specifically, the flow configuration is based on the one presented in Schäfer et al. (1996) and features a cylinder immersed in a 2D incompressible flow injected following a parabolic velocity profile. The Reynolds number based on the cylinder diameter is moderate to keep the CFD affordable, and following Schäfer et al. (1996), Re=100 is used. The incremental pressure correction scheme is used together with a finite element approach to solve the discretized problem. A periodic Karman vortex alley is then obtained, as visible in figure 8. Besides, small jets are added on the sides of the cylinder in order to allow controlling the separation of the wake. Therefore, the action performed by the DRL agent is to decide the instantaneous mass flow rate of the small jets, while the reward is chosen to encourage drag reduction. Finally, probes reporting the characteristics of the flow at several fixed points in the computational domain (either pressure or velocity can be used, both providing equivalent results) are added in the vicinity of the cylinder and are used as the observation provided as input to the network.
Good results are obtained, with training performed in around 1300 vortex shedding periods. The control strategy is more complicated than traditional harmonic forcing that had been performed in previous works (Bergmann et al., 2005), which illustrates the value of using an ANN as the controller. Typically, around 93% of the drag induced by the vortex shedding effect (Bergmann et al., 2005) is suppressed by the control law found, and the strength of the jets needed to reduce the drag is minimal. In the established regime, the magnitude of the mass flow rate injected by the jets normalized by the mass flow rate of the base flow intersecting the cylinder is only of typically 0.6%. The resulting flow, visible in figure 8, is similar to what would be obtained with boat tailing; more specifically, an apparent increase in the recirculation bubble is observed.
The work of Rabault et al. (2019) shows that active flow control is, at least on a simple flow configuration, achievable with small actuations if an adapted control law is used. While there are still many unanswered questions at the moment, such as the robustness of the control law, the application of the methodology to higher, more realistic (from an industrial point of view) Reynolds numbers, or the possibility to control 3D flows, this work establishes DRL as a new methodology that should be further investigated for the task of performing active flow control.
3.6 Accelerating deep reinforcement learning of active flow control strategies through a multi environment approach (Rabault & Kuhnle, 2019)
One of the main practical limitations of Rabault et al. (2019) lies within the time needed to perform the learning  around 24 hours on a modern CPU. This comes from the inherent cost of the CFD, and the fact that it is challenging to obtain computational speedups on small tasks. In particular, increasing the number of cores used for solving the CFD problem of Rabault et al. (2019) provides very little speedup, as the communication between the cores negates other speed gains. While this is particularly true on such a small computational problem, any CFD simulation will only speed up to a maximum point independently of the number of cores used. Therefore, a solution is needed to provide the data necessary for the ANN to perform learning in a reasonable amount of time.
In Rabault & Kuhnle (2019), the collection of data used for filling the memory replay buffer is parallelized by using an ensemble of simulations all running in parallel, independently of each other. In practice, this means that the knowledge obtained from several CFD simulations running in an embarrassingly parallel fashion can be focused on the same ANN to speed up its learning.
This technique results in large speedups for the learning process, as visible in figure 9. In particular, for a number of simulations that are a divider of the learning frequency of the ANN, perfect scaling is obtained, and the training is formally equivalent to the one obtained in serial (i.e. , with one single simulation). However, even more simulations can be used, and it is found empirically that the resulting offpolicy data sampling does not sensibly reduce the learning ability, as visible in figure 9. This means in practice that the authors achieve a training speedup factor of up to around 60.
This natural parallelism of the learning process of ANNs trained through DRL represents a significant advantage for this methodology, and a promising technical result that opens the way to the control of much more sophisticated situations.
3.7 Sensitivity of aerodynamic forces in laminar flows past a square cylinder
In this section, we propose an original comparison between a classical optimization approach and DRL. The test case is inspired by Meliga et al. (2014), where two cylinders are immersed in a moderate Reynolds flow. One is a square main cylinder, which position is fixed, while the other is a cylindrical control cylinder, which position can vary, and which radius is considered much smaller than that of the main cylinder. The goal of this application is to optimize the position of the small cylinder, such that the total drag of the two cylinders is inferior to that of the main cylinder alone. An incompressible NavierStokes solver Alnæs et al. (2015) is used to simulate the flow at different Reynolds number past the square cylinder. Moreover, the flow conditions are identical to that described in Rabault et al. (2019). Six different agents are trained in different conditions, as detailed in table 3.
Agent  $Re$  Objective  Details 

1.1  10  Control over $17$ time steps  Trained from sratch 
1.2  10  Direct optimization  Trained from sratch 
2.1  40  Control over $17$ time steps  Transfer learning from 1.1 
2.2  40  Direct optimization  Transfer learning from 1.2 
3.1  100  Control over $17$ time steps  Transfer learning from 1.1 
3.2  100  Direct optimization  Transfer learning from 1.2 
Two different architectures are proposed. The first one is a classical DRL architecture, similar to what is shown in figure 1, but with a difference that episodes were reduced to a single action. In that case, the agent directly attempts to propose an optimal $(x,y)$ position from the same initial state, using a PPO algorithm. On the contrary, in the second architecture, successive displacements $(\mathrm{\Delta}x,\mathrm{\Delta}y)$ of the control cylinder are provided by an A3C agent, starting from a random initial position. The observations provided to the agent are made of the last two positions of the control cylinder, the Reynolds number and encoded features from the velocity fields. To that end, an autoencoder was trained simultaneously to the agent to reduce the state dimensionality (with a dimension reduction from around 8000 to 70), in a similar fashion to Ma et al. (2018) (see figure 10). The same reward function is used for both architectures:
$${r}_{t}={C}_{D}^{\mathrm{\hspace{0.17em}0}}{C}_{D},$$ 
where ${C}_{D}^{\mathrm{\hspace{0.17em}0}}$ is the drag of the main cylinder alone, ${C}_{D}$ is the total drag of the main and the control cylinders, and the drag coefficients are counted positively. With this simple expression, a positive reward is obtained when the combined drag of the cylinders is inferior to the reference drag value.
We compare the results obtained with those from Meliga et al. (2014), where a classical adjoint method is used. Overall, the same optimal positions were found for the control cylinder, at $Re=40$ and $Re=100$. Although the case $Re=10$ was not present in the original paper, coherent results were obtained for the configuration too. For the first agents at $Re=10$, training took approximately $5$ hours using parallel learning with 17 CPUs. Regarding the last four agents, the transfer learning allowed them to learn at a much faster pace: only $10$ hours were necessary on a single CPU. An overview of the results is shown in figure 11.
With this experiment, we showed that not only DRL could perform both control and direct optimization tasks, but also that transfer learning between different configurations represents a powerful feature to save computational time. We also confirmed that the use of an autoencoder is a safe and robust way to extract essential features from complex and high dimensional fluid fields. Details about this test case and the code used to reproduce these results can be found on the GitHub repository: https://github.com/DonsetPG/fenicsDRL. We recall that this code uses different library:
Future work on this library will focus on generalizing the coupling of DRL with different open source CFD code.
4 Conclusion
Although DRL has already been applied to several cases of optimization and control in the context of fluid dynamics, the literature on the topic remains particularly shallow. In the present article, we reviewed the available contributions on the topic to provide the reader with a comprehensive state of play of the possibilities of DRL in fluid mechanics. In each of them, details were provided on the numerical context and the problem complexity. The choices of the DRL algorithm and the reward shaping were also described.
Given the highlevel interfaces of existing libraries, the coupling of DRL algorithms with existing numerical CFD solvers can be achieved with a minimal investment, while opening a wide range of possibilities in terms of optimization and control tasks. The algorithms presented in this review proved to be robust when exposed to possible numerical noise, although high $Re$ applications remain to be achieved. Additionally, the parallel capabilities of the main DRL libraries represent a major asset in the context of timeexpensive CFD computations. Transfer learning also proved to be a key feature in saving computational time, as retraining agents already exploited in similar situations led to a fast convergence of the agent policy. The use of autoencoders to feed the agent with both a compact and rich observation of the environment also showed to be beneficial, as it allows for a reduced size of the agent network, thus implying a faster convergence.
As of now, the capabilities and robustness of DRL algorithms in highly turbulent and nonlinear flows remain to be explored. Also, their behavior and convergence speed in action spaces of high dimensionalities is unknown. It makes no doubt that the upcoming years will see the mastering of these obstacles, supported by the constant progress made in the DRL field and driven by the numerous industrial challenges that could benefit from it.
References
 of Aeronautics & Astronautics (2008) of Aeronautics, American Institute & Astronautics, ed. 2008 Morphing Airfoils with Four Morphing Parameters.
 Ali & Behdinan (2003) Ali, N & Behdinan, K 2003 Optimal geometrical design of aircraft using genetic algorithms. Transactions of the Canadian Society for Mechanical Engineering 26, 373–388.
 Alnæs et al. (2015) Alnæs, M.S., Bletcha, J, Hake, A, Johansson, B, Kehlet, B, Logg, A, Richardson, C., Ring, J., Rognes, M.E. & Wells, G.N. 2015 The FEniCS Project Version 1.5. Archive of Numerical Software 3.
 Bahdanau et al. (2016) Bahdanau, Dzmitry, Brakel, Philemon, Xu, Kelvin, Goyal, Anirudh, Lowe, Ryan, Pineau, Joelle, Courville, Aaron & Bengio, Yoshua 2016 An ActorCritic Algorithm for Sequence Prediction. CoRR abs/1607.07086 (2015), 1–17.
 Bellman (1957) Bellman, R. 1957 A Markovian Decision Process. Journal of Mathematics and Mechanics 6 (5), 679–684.
 Bellman & Dreyfus (1962) Bellman, R. & Dreyfus, S. E. 1962 Applied dynamic programming. Princeton University Press Princeton, N.J.
 Bergmann et al. (2005) Bergmann, Michel, Cordier, Laurent & Brancher, JeanPierre 2005 Optimal rotary control of the cylinder wake using proper orthogonal decomposition reducedorder model. Physics of fluids 17 (9), 097101.
 Bewley et al. (2018) Bewley, Alex, Rigley, Jessica, Liu, Yuxuan, Hawke, Jeffrey, Shen, Richard, Lam, VinhDieu & Kendall, Alex 2018 Learning to Drive from Simulation without Real World Labels. arXiv eprints , arXiv: 1812.03823.
 Brockman et al. (2016) Brockman, Greg, Cheung, Vicki, Pettersson, Ludwig, Schneider, Jonas, Schulman, John, Tang, Jie & Zaremba, Wojciech 2016 OpenAI Gym. arXiv eprints .
 Brown & Sandholm (2019) Brown, Noam & Sandholm, Tuomas 2019 Superhuman AI for multiplayer poker. Science , arXiv: https://science.sciencemag.org/content/early/2019/07/10/science.aay2400.
 Bucci et al. (2019) Bucci, Michele Alessandro, Semeraro, Onofrio, Allauzen, Alexandre, Wisniewski, Guillaume, Cordier, Laurent & Mathelin, Lionel 2019 Control of chaotic systems by Deep Reinforcement Learning. arXiv p. arXiv:1906.07672.
 Colabrese et al. (2017) Colabrese, Simona, Gustavsson, Kristian, Celani, Antonio & Biferale, Luca 2017 Flow Navigation by Smart Microswimmers via Reinforcement Learning. Physical Review Letters 118 (15), 158004.
 Françoislavet et al. (2018) Françoislavet, Vincent, Henderson, Peter, Islam, Riashat & Bellemare, Marc G 2018 An Introduction to Deep Reinforcement Learning. Foundations and trends in machine learning .
 Garnier & Viquerat (2019) Garnier, Paul & Viquerat, Jonathan 2019 Position of a control cylinder to reduce drag. https://github.com/DonsetPG/fenicsDRL.
 Gazzola et al. (2016) Gazzola, M., Tchieu, A. A., Alexeev, D., de Brauer, A. & Koumoutsakos, P. 2016 Learning to school in the presence of hydrodynamic interactions. Journal of Fluid Mechanics 789, 726–749.
 Goodfellow et al. (2017) Goodfellow, Ian, Bengio, Yoshua & Courville, Aaron 2017 The Deep Learning Book. MIT Press.
 Guéniat et al. (2016) Guéniat, Florimond, Mathelin, Lionel & Hussaini, M. Yousuff 2016 A statistical learning strategy for closedloop control of fluid flows. Theoretical and Computational Fluid Dynamics 30 (6), 497–510.
 Gustavsson et al. (2017) Gustavsson, K., Biferale, L., Celani, A. & Colabrese, S. 2017 Finding Efficient Swimming Strategies in a Three Dimensional Chaotic Flow by Reinforcement Learning. arXiv eprints , arXiv: 1711.05826.
 Hill et al. (2018) Hill, Ashley, Raffin, Antonin, Ernestus, Maximilian, Gleave, Adam, Traore, Rene, Dhariwal, Prafulla, Hesse, Christopher, Klimov, Oleg, Nichol, Alex, Plappert, Matthias, Radford, Alec, Schulman, John, Sidor, Szymon & Wu, Yuhuai 2018 Stable Baselines. https://github.com/hilla/stablebaselines.
 Hochreiter & Schmidhuber (1997) Hochreiter, Sepp & Schmidhuber, Jürgen 1997 Long shortterm memory. Neural Comput 9, "1735–1780".
 Hou Tsang et al. (2018) Hou Tsang, Alan Cheng, Tong, Pun Wai, Nallan, Shreyes & Pak, On Shun 2018 Selflearning how to swim at low Reynolds number. arXiv p. arXiv:1808.07639.
 Houthooft et al. (2018) Houthooft, Rein, Chen, Richard Y., Isola, Phillip, Stadie, Bradly C., Wolski, Filip, Ho, Jonathan & Abbeel, Pieter 2018 Evolved Policy Gradients. arXiv eprints .
 Howard (1960) Howard, R. A. 1960 Dynamic programming and Markov processes. Technology Press and Wiley, New York.
 Kendall et al. (2018) Kendall, Alex, Hawke, Jeffrey, Janz, David, Mazur, Przemyslaw, Reda, Daniele, Allen, JohnMark, Lam, VinhDieu, Bewley, Alex & Shah, Amar 2018 Learning to Drive in a Day. arXiv eprints , arXiv: 1807.00412.
 Knight (2018) Knight, Will 2018 Google just gave control over data center cooling to an AI. https://www.technologyreview.com/s/611902/googlejustgavecontroloverdatacentercoolingtoanai/.
 Lee & Hajela (1996) Lee, J & Hajela, P 1996 Parallel genetic algorithm implementation in multidisciplinary rotor blade design. Journal of Aircraft 33, 962–969.
 Lee et al. (2018) Lee, Xian Yeow, Balu, Aditya, Stoecklein, Daniel, Ganapathysubramanian, Baskar & Sarkar, Soumik 2018 Flow Shape Design for Microfluidic Devices Using Deep Reinforcement Learning. CoRR abs/1811.12444, 1–10.
 Lillicrap et al. (2015) Lillicrap, Timothy P., Hunt, Jonathan J., Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David & Wierstra, Daan 2015 Continuous control with deep reinforcement learning. arXiv eprints .
 Lin (1993) Lin, LongJi 1993 Reinforcement Learning for Robots Using Neural Networks. PhD thesis, Carnegie Mellon University, Pittsburgh, PA, USA, uMI Order No. GAX9322750.
 Liou et al. (2014) Liou, ChengYuan, Cheng, WeiChen, Liou, JiunWei & Liou, DawRan 2014 Autoencoder for words. Neurocomputing 139, 84–96.
 Ma et al. (2018) Ma, Pingchuan, Tian, Yunsheng, Pan, Zherong, Ren, Bo & Manocha, Dinesh 2018 Fluid directed rigid body control using deep reinforcement learning. ACM Transactions on Graphics 37, 1–11.
 Mäkinen et al. (1999) Mäkinen, R.A.E, Periaux, J & Toivanen, J 1999 Multidisciplinary shape optimization in aerodynamics and electromagnetics using genetic algorithms. International Journal for Numerical Methods in Fluids 30, 149–159.
 Matos et al. (2004) Matos, R, Laursen, T, Vargas, J & Bejan, A 2004 Threedimensional optimization of staggered finned circular and elliptic tubes in forced convection. International Journal of Thermal Sciences 43, 477–487.
 Meliga et al. (2014) Meliga, Philippe, Boujo, Edouard, Pujals, Gregory & Gallaire, François 2014 Sensitivity of aerodynamic forces in laminar and turbulent flow past a square cylinder. Physics of Fluids 26, 26,104101.
 Mnih et al. (2013) Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Graves, Alex, Antonoglou, Ioannis, Wierstra, Daan & Riedmiller, Martin 2013 Playing Atari with Deep Reinforcement Learning. CoRR abs/1312.5602.
 Mnih et al. (2016) Mnih, Volodymyr, Puigdomènech Badia, Adrià, Mirza, Mehdi, Graves, Alex, Lillicrap, Timothy P., Harley, Tim, Silver, David & Kavukcuoglu, Koray 2016 Asynchronous Methods for Deep Reinforcement Learning. arXiv eprints , arXiv: 1602.01783.
 Muyl et al. (2004) Muyl, F, Dumas, L & Herbert, V 2004 Hybrid method for aerodynamic shape optimization in automotive industry. Computers & Fluids 33, 849–858.
 Novati et al. (2017) Novati, Guido, Verma, Siddhartha, Alexeev, Dmitry, Rossinelli, Diego, van Rees, Wim M. & Koumoutsakos, Petros 2017 Synchronised swimming of two fish. Bioinspiration and Biomimetics 12 (3), 036001.
 OpenAI (2018) OpenAI 2018 OpenAI Five. https://blog.openai.com/openaifive/.
 Pinto et al. (2017) Pinto, Lerrel, Andrychowicz, Marcin, Welinder, Peter, Zaremba, Wojciech & Abbeel, Pieter 2017 Asymmetric Actor Critic for ImageBased Robot Learning. CoRR abs/1710.06542.
 Qiu et al. (2018) Qiu, Jingran, Zhao, Lihao, Xu, Chunxiao & Yao, Yichen 2018 Swimming strategy of settling elongated microswimmers by reinforcement learning. arXiv p. arXiv:1811.10880.
 Rabault et al. (2019) Rabault, Jean, Kuchta, Miroslav, Jensen, Atle, Réglade, Ulysse & Cerardi, Nicolas 2019 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. Journal of Fluid Mechanics 865, 281–302.
 Rabault & Kuhnle (2019) Rabault, Jean & Kuhnle, Alexander 2019 Accelerating Deep Reinforcement Learning of Active Flow Control strategies through a multienvironment approach. arXiv eprints , arXiv: 1906.10382.
 Schäfer et al. (1996) Schäfer, Michael, Turek, Stefan, Durst, Franz, Krause, Egon & Rannacher, Rolf 1996 Benchmark computations of laminar flow around a cylinder. In Flow simulation with HighPerformance Computers II, pp. 547–566. Springer.
 Schaul et al. (2015) Schaul, Tom, Quan, John, Antonoglou, Ioannis & Silver, David 2015 Prioritized Experience Replay. arXiv eprints , arXiv: 1511.05952.
 Schulman et al. (2015) Schulman, John, Levine, Sergey, Moritz, Philipp, Jordan, Michael I. & Abbeel, Pieter 2015 Trust Region Policy Optimization. arXiv eprints , arXiv: 1502.05477.
 Schulman et al. (2016) Schulman, John, Moritz, Philipp, Levine, Sergey, Jordan, Michael & Abbeel, Pieter 2016 HighDimensional Continuous Control Using Generalized Advantage Estimation. arXiv eprints pp. 1–14.
 Siegelmann & Sontag (1995) Siegelmann, H.T. & Sontag, E.D. 1995 On the Computational Power of Neural Nets. Journal of Computer and System Sciences 50 (1), 132–150.
 Silver et al. (2017) Silver, David, Schrittwieser, Julian, Simonyan, Karen, Antonoglou, Ioannis, Huang, Aja, Guez, Arthur, Hubert, Thomas, Baker, Lucas, Lai, Matthew, Bolton, Adrian, Chen, Yutian, Lillicrap, Timothy, Hui, Fan, Sifre, Laurent, van den Driessche, George, Graepel, Thore & Hassabis, Demis 2017 Mastering the game of Go without human knowledge. Nature 550.
 Sutton (1988) Sutton, R.S 1988 Learning to Predict by the Method of Temporal Differences. Machine Learning 3 (1), 9–44.
 Sutton & Barto (1998) Sutton, R. S. & Barto, A. G. 1998 Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA.
 Sutton & Barto (2018) Sutton, R. S. & Barto, A. G. 2018 Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA.
 Tesauro (1995) Tesauro, Gerald 1995 Temporal Difference Learning and TDGammon. Communications of the ACM 38.
 Tsitsiklis & Van Roy (1997) Tsitsiklis, John & Van Roy, Benjamin 1997 An analysis of temporaldifference learning with function approximation. IEEE Transactions p. 674–690.
 van Hasselt et al. (2015) van Hasselt, Hado, Guez, Arthur & Silver, David 2015 Deep Reinforcement Learning with Double Qlearning. arXiv eprints , arXiv: 1509.06461.
 Verma et al. (2018) Verma, Siddhartha, Novati, Guido & Koumoutsakos, Petros 2018 Efficient collective swimming by harnessing vortices through deep reinforcement learning. arXiv eprints , arXiv: 1802.02674.
 Vinyals et al. (2019) Vinyals, Oriol, Babuschkin, Igor, Chung, Junyoung, Mathieu, Michael, Jaderberg, Max, Czarnecki, Wojciech M., Dudzik, Andrew, Huang, Aja, Georgiev, Petko, Powell, Richard, Ewalds, Timo, Horgan, Dan, Kroiss, Manuel, Danihelka, Ivo, Agapiou, John, Oh, Junhyuk, Dalibard, Valentin, Choi, David, Sifre, Laurent, Sulsky, Yury, Vezhnevets, Sasha, Molloy, James, Cai, Trevor, Budden, David, Paine, Tom, Gulcehre, Caglar, Wang, Ziyu, Pfaff, Tobias, Pohlen, Toby, Wu, Yuhuai, Yogatama, Dani, Cohen, Julia, McKinney, Katrina, Smith, Oliver, Schaul, Tom, Lillicrap, Timothy, Apps, Chris, Kavukcuoglu, Koray, Hassabis, Demis & Silver, David 2019 AlphaStar: Mastering the RealTime Strategy Game StarCraft II. https://deepmind.com/blog/alphastarmasteringrealtimestrategygamestarcraftii/.