Conservation laws are considered to be fundamental laws of nature. It hasbroad applications in many fields, including physics, chemistry, biology,geology, and engineering. Solving the differential equations associated withconservation laws is a major branch in computational mathematics. The recentsuccess of machine learning, especially deep learning in areas such as computervision and natural language processing, has attracted a lot of attention fromthe community of computational mathematics and inspired many intriguing worksin combining machine learning with traditional methods. In this paper, we arethe first to view numerical PDE solvers as an MDP and to use (deep) RL to learnnew solvers. As proof of concept, we focus on 1-dimensional scalar conservationlaws. We deploy the machinery of deep reinforcement learning to train a policynetwork that can decide on how the numerical solutions should be approximatedin a sequential and spatial-temporal adaptive manner. We will show that theproblem of solving conservation laws can be naturally viewed as a sequentialdecision-making process, and the numerical schemes learned in such a way caneasily enforce long-term accuracy. Furthermore, the learned policy network iscarefully designed to determine a good local discrete approximation based onthe current state of the solution, which essentially makes the proposed methoda meta-learning approach. In other words, the proposed method is capable oflearning how to discretize for a given situation mimicking human experts.Finally, we will provide details on how the policy network is trained, how wellit performs compared with some state-of-the-art numerical solvers such as WENOschemes, and supervised learning based approach L3D and PINN, and how well itgeneralizes.
Quick Read (beta)
Learning to Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning
Conservation laws are considered to be fundamental laws of nature. It has broad applications in many fields, including physics, chemistry, biology, geology, and engineering. Solving the differential equations associated with conservation laws is a major branch in computational mathematics. The recent success of machine learning, especially deep learning in areas such as computer vision and natural language processing, has attracted a lot of attention from the community of computational mathematics and inspired many intriguing works in combining machine learning with traditional methods. In this paper, we are the first to view numerical PDE solvers as an MDP and to use (deep) RL to learn new solvers. As proof of concept, we focus on 1-dimensional scalar conservation laws. We deploy the machinery of deep reinforcement learning to train a policy network that can decide on how the numerical solutions should be approximated in a sequential and spatial-temporal adaptive manner. We will show that the problem of solving conservation laws can be naturally viewed as a sequential decision-making process, and the numerical schemes learned in such a way can easily enforce long-term accuracy. Furthermore, the learned policy network is carefully designed to determine a good local discrete approximation based on the current state of the solution, which essentially makes the proposed method a meta-learning approach. In other words, the proposed method is capable of learning how to discretize for a given situation mimicking human experts. Finally, we will provide details on how the policy network is trained, how well it performs compared with some state-of-the-art numerical solvers such as WENO schemes, and supervised learning based approach L3D and PINN, and how well it generalizes.
Wang Y F et. al.]Yufei Wang 22footnotemark: 2 ††footnotetext: The first two authors contributed equally., Ziju Shen , Zichao Long and Bin Dong11 1 Corresponding author.
Learning to Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning
Computer Science Department, Carnegie Mellon University, USA.
Academy for Advanced Interdisciplinary Studies, Peking University, P.R. China.
School of Mathematical Sciences, Peking University, P.R. China.
Beijing International Center for Mathematical Research, and Institute for Artificial Intelligence, Peking University, P.R. China.
AMS subject classifications: 65M06, 68T05, 49N90
Key words: conservation laws, deep reinforcement learning, finite difference approximation, WENO
Conservation laws are considered to be one of the fundamental laws of nature, and has broad applications in multiple fields such as physics, chemistry, biology, geology, and engineering. For example, Burgers equation, a very classic partial differential equation (PDE) in conservation laws, has important applications in fluid mechanics, nonlinear acoustics, gas dynamics, and traffic flow.
Solving the differential equations associated with conservation laws has been a major branch of computational mathematics [18, 19], and a lot of effective methods have been proposed, from classic methods such as the upwind scheme, the Lax-Friedrichs scheme, to the advanced ones such as the ENO/WENO schemes [22, 30], the flux-limiter methods , and etc. In the past few decades, these traditional methods have been proven successful in solving conservation laws. Nonetheless, the design of some of the high-end methods heavily relies on expert knowledge and the coding of these methods can be a laborious process. To ease the usage and potentially improve these traditional algorithms, machine learning, especially deep learning, has been recently incorporated into this field. For example, the ENO scheme requires lots of ‘if/else’ logical judgments when used to solve complicated system of equations or high-dimensional equations. This very much resembles the old-fashioned expert systems. The recent trend in artificial intelligence (AI) is to replace the expert systems by the so-called ‘connectionism’, e.g., deep neural networks, which leads to the recent bloom of AI. Therefore, it is natural and potentially beneficial to introduce deep learning in traditional numerical solvers of conservation laws.
1.1 Related works
In the last few years, neural networks (NNs) have been applied to solving ODEs/PDEs or the associated inverse problems. These works can be roughly classified into two categories according to the way that the NN is used.
The first type of works propose to harness the representation power of NNs, and are irrelevant to the numerical discretization based methods. For example, in the pioneering works [34, 31, 27, 26, 2], NNs are used as new ansatz to approximate solutions of PDEs. It was later generalized by  to allow randomness in the solution which is trained using policy gradient. More recent works along this line include [23, 25, 6]. Besides, several works have focused on using NNs to establish direct mappings between the parameters of the PDEs (e.g. the coefficient field or the ground state energy) and their associated solutions [16, 17, 21, 9]. Furthermore, [13, 4] proposed a method to solve very high-dimensional PDEs by converting the PDE to a stochastic control problem and use NNs to approximate the gradient of the solution.
The second type of works, which target at using NNs to learn new numerical schemes, are closely related to our work. However, we note that these works mainly fall in the setting of supervised learning (SL). For example,  proposed to integrate NNs into high-order numerical solvers to predict artificial viscosity;  trained a multilayer perceptron to replace traditional indicators for identifying troubled-cells in high-resolution schemes for conservation laws. These works greatly advanced the development in machine learning based design of numerical schemes for conservation laws. Note that in , the authors only utilized the one-step error to train the artificial viscosity networks without taking into account the long-term accuracy of the learned numerical scheme.  first constructed several functions with known regularities and then used them to train a neural network to predict the location of discontinuity, which was later used to choose a proper slope limiter. Therefore, the training of the NNs is separated from the numerical scheme. Then, a natural question is whether we can learn discretization of differential equations in an end-to-end fashion and the learned discrete scheme also takes long-term accuracy into account. This motivates us to employ reinforcement learning to learn good solvers for conservation laws.
1.2 Our Approach
The main objective of this paper is to design new numerical schemes in an autonomous way. We propose to view iterative numerical schemes solving the conservation laws as a Markov Decision Process (MDP) and use reinforcement learning (RL) to find new and potentially better data-driven solvers. We carefully design an RL-based method so that the learned policy can generate high accuracy numerical schemes and can well generalize in varied situations. Details will be given in section 3.
Here, we first summarize the benefits and potentials of using RL to solve conservation laws (the arguments may also well apply to other evolution PDEs):
Most of the numerical solvers of conservation law can be interpreted naturally as a sequential decision making process (e.g., the approximated grid values at the current time instance definitely affects all the future approximations). Thus, it can be easily formulated as a MDP and solved by RL.
In almost all the RL algorithms, the policy (which is the AI agent who decides on how the solution should be approximated locally) is optimized with regards to the values , which by definition considers the long-term accumulated reward (or, error of the learned numerical scheme), thus could naturally guarantee the long-term accuracy of the learned schemes, instead of greedily deciding the local approximation which is the case for most numerical PDEs solvers. Furthermore, it can gracefully handle the cases when the action space is discrete, which is in fact one of the major strength of RL.
By optimizing towards long-term accuracy and effective exploration, we believe that RL has a good potential in improving traditional numerical schemes, especially in parts where no clear design principles exist. For example, although the WENO-5 scheme achieves optimal order of accuracy at smooth regions of the solution , the best way of choosing templates near singularities remains unknown. Our belief that RL could shed lights on such parts is later verified in the experiments: the trained RL policy demonstrated new behaviours and is able to select better templates than WENO and hence approximate the solution better than WENO near singularities.
Non-smooth norms such as the infinity norm of the error is often used to evaluate the performance of the learned numerical schemes. As the norm of the error serves as the loss function for the learning algorithms, computing the gradient of the infinity norm can be problematic for supervised learning, while RL does not have such problem since it does not explicitly take gradients of the loss function (i.e. the reward function for RL).
Learning the policy within the RL framework makes the algorithm meta-learning-like [29, 5, 1, 20, 10]. The learned policy can decide on which local numerical approximation to use by judging from the current state of the solution (e.g. local smoothness, oscillatory patterns, dissipation, etc). This is vastly different from regular (non-meta-) learning where the algorithms directly make inference on the numerical schemes without the aid of an additional network such as . As subtle the difference as it may seem, meta-learning-like methods have been proven effective in various applications such as in image restoration [15, 8, 35]. See  for a comprehensive survey on meta-learning. In our experiment section, we will present extensive empirical studies on the generalization ability of the RL-based solver and demonstrate its advantage over some supervised learning based methods.
Another purpose of this paper is to raise an awareness of the connection between MDP and numerical PDE solvers, and the general idea of how to use RL to improve PDE solvers or even finding brand new ones. Furthermore, in computational mathematics, a lot of numerical algorithms are sequential, and the computation at each step is expert-designed and usually greedy, e.g., the conjugate gradient method, the fast sweeping method , matching pursuit , etc. We hope our work could motivate more researches in combining RL and computational mathematics, and stimulate more exploration on using RL as a tool to tackle the bottleneck problems in computational mathematics.
Our paper is organized as follows. In section 2 we briefly review 1-dimensional conservation laws and the WENO schemes. In section 3, we discuss how to formulate the process of numerically solving conservation laws into a Markov Decision Process. Then, we present details on how to train a policy network to mimic human expert in choosing discrete schemes in a spatial-temporary adaptive manner by learning upon WENO. In section 4, we conduct numerical experiments on 1-D conservation laws to demonstrate the performance of our trained policy network. Our experimental results show that the trained policy network indeed learned to adaptively choose good discrete schemes that offer better results than the state-of-the-art WENO scheme which is 5th order accurate in space and 4th order accurate in time. This serves as an evidence that the proposed RL framework has the potential to design high-performance numerical schemes for conservation laws in a data-driven fashion. Furthermore, the learned policy network generalizes well to other situations such as different initial conditions, forcing terms, mesh sizes, temporal discrete schemes, etc. To further demonstrate the advantage of using RL to design numerical solvers, we we provide additional comparisons between the WENO learned using RL and two supervised learning based methods, L3D  and PINN [27, 26]. The paper ends with a conclusion in section 5, where possible future research directions are also discussed.
In this paper, we consider solving the following 1-D conservation laws:
For example, is the famous Burgers Equation. We discretize the -plane by choosing a mesh with spatial size and temporal step size , and define the discrete mesh points by
We denote The finite difference methods will produce approximations to the solution on the given discrete mesh points. We denote point-wise values of the true solution to be , and the true point-wise flux values to be .
2.2 WENO – Weighted Essentially Non-Oscillatory Schemes
WENO (Weighted Essentially Non-Oscillatory)  is a family of high order accurate finite difference schemes for solving hyperbolic conservation laws, and has been successful for many practical problems. The key idea of WENO is a nonlinear adaptive procedure that automatically chooses the smoothest local stencil to reconstruct the numerical flux. Generally, a finite difference method solves Eq.2.1 by using a conservative approximation to the spatial derivative of the flux:
where is the numerical approximation to the point value and is the numerical flux generated by a numerical flux policy
which is manually designed. Note that the term “numerical flux policy" is a new terminology that we introduce in this paper, which is exactly the policy we shall learn using RL. In WENO, works as follows. Using the physical flux values , we could obtain a order accurate polynomial interpolation , where the indices is called a ‘stencil’. We could also use the stencil , or to obtain another three interpolants , and . The key idea of WENO is to average (with properly designed weights) all these interpolants to obtain the final reconstruction:
The weight depends on the smoothness of the stencil. A general principal is: the smoother is the stencil, the more accurate is the interpolant and hence the larger is the weight. To ensure convergence, we need the numerical scheme to be consistent and stable . It is known that WENO schemes as described above are consistent. For stability, upwinding is required in constructing the flux. The most easy way is to use the sign of the Roe speed
to determine the upwind direction: if , we only average among the three interpolants , and ; if , we use , and .
Some further thoughts.
WENO achieves optimal order of accuracy (up to 5) at the smooth region of the solutions , while lower order of accuracy at singularities. The key of the WENO method lies in how to compute the weight vector , which primarily depends on the smoothness of the solution at local stencils. In WENO, such smoothness is characterized by handcrafted formula, and was proven to be successful in many practical problems when coupled with high-order temporal discretization. However, it remains unknown whether there are better ways to combine the stencils so that optimal order of accuracy in smooth regions can be reserved while, at the same time, higher accuracy can be achieved near singularities. Furthermore, estimating the upwind directions is another key component of WENO, which can get quite complicated in high-dimensional situations and requires lots of logical judgments (i.e. “if/else"). Can we ease the (some time painful) coding and improve the estimation at the aid of machine learning?
In this section we present how to employ reinforcement learning to solve the conservation laws given by (2.1). To better illustrate our idea, we first show in general how to formulate the process of numerically solving a conservation law into an MDP. We then discuss how to incorporate a policy network with the WENO scheme. Our policy network targets at the following two key aspects of WENO: (1) Can we learn to choose better weights to combine the constructed fluxes? (2) Can we learn to automatically judge the upwind direction, without complicated logical judgments?
3.1 MDP Formulation
As shown in Algorithm 1, the procedure of numerically solving a conservation law is naturally a sequential decision making problem. The key of the procedure is the numerical flux policy and the temporal scheme as shown in line 6 and 8 in Algorithm 1. Both policies could be learned using RL. However, in this paper, we mainly focus on using RL to learn the numerical flux policy , while leaving the temporal scheme with traditional numerical schemes such as the Euler scheme or the Runge–Kutta methods. A quick review of RL is given in the appendix.
Now, we show how to formulate the above procedure as an MDP and the construction of the state , action , reward and transition dynamics . Algorithm 2 shows in general how RL is incorporated into the procedure. In Algorithm 2, we use a single RL agent. Specifically, when computing :
The state for the RL agent is
where is the state function. The choice of depends on the problem and the choice we made for WENO will be given in Section 4.
In general, the action of the agent is used to determine how the numerical fluxes and is computed. In the next subsection, we detail how we incorporate to be the linear weights of the fluxes computed using different stencils in the WENO scheme.
The reward should encourage the agent to generate a scheme that minimizes the error between its approximated value and the true value. Therefore, we define the reward function as
e.g., a simplest choice is . Same as , the choice of is problem dependent, and the choice we made for WENO will be given in Section 4.
The transition dynamics is fully deterministic, and depends on the choice of the temporal scheme at line 10 in Algorithm 2. Note that the next state can only be constructed when we have obtained all the point values in the next time step, i.e., does not only depends on action , but also on actions (action can only determine the value ). This subtlety can be resolved by viewing the process under the framework of multi-agent RL, in which at each mesh point we use a distinct agent , and the next state depends on these agents’ joint action . However, it is impractical to train different agents as is usually very large, therefore we enforce the agents at different mesh point to share the same weight, which reduces to case of using just a single agent. The single agent can be viewed as a counterpart of a human designer who decides on the choice of a local scheme based on the current state in traditional numerical methods.
3.2 Improving WENO with RL
We now present how to transfer the actions of the RL policy to the weights of WENO fluxes. Instead of directly using to generate the numerical flux, we use it to produce the weights of numerical fluxes computed using different stencils in WENO. Since the weights are part of the configurations of the WENO scheme, our design of action essentially makes the RL policy a meta-learner, and enables more stable learning and better generalization power than directly generating the fluxes.
Specifically, at point (here we drop the time superscript for simplicity), to compute the numerical flux and , we first construct four fluxes and using four different stencils just as in WENO, and then use the RL policy to generate the weights of these fluxes:
The numerical flux is then constructed by averaging these fluxes:
Note that the determination of upwind direction is automatically embedded in the RL policy since it generates four weights at once. For instance, when the roe speed , we expect the weight and when , we expect . Note that the upwind direction can be very complicated in a system of equations or in the high-dimensional situations, and using the policy network to automatically embed such a process could save lots of efforts in algorithm design and implementation. Our numerical experiments show that can indeed automatically determine upwind directions for 1D scalar cases. Although this does not mean that it works for systems and/or in high-dimensions, it shows the potential of the proposed framework and value for further studies.
|0.002||10.81 (3.96)||11.13 (3.83)||18.79 (12.44)||19.45 (9.32)||28.61 (33.37)||35.95 (27.7)|
|0.003||10.83 (3.96)||11.14 (3.82)||18.82 (12.48)||19.47 (9.31)||28.62 (33.34)||35.96 (27.67)|
|0.004||10.83 (3.96)||11.14 (3.84)||18.89 (12.69)||19.48 (9.33)||28.61 (33.27)||35.93 (27.63)|
|0.005||10.85 (3.96)||11.15 (3.84)||18.96 (12.84)||19.52 (9.35)||28.48 (33.04)||35.93 (27.61)|
|0.006||10.89 (3.93)||11.16 (3.83)||18.95 (12.79)||19.51 (9.3)||28.58 (33.08)||35.89 (27.5)|
|0.002||10.05 (2.74)||10.25 (2.65)||16.89 (3.59)||17.09 (3.56)||17.07 (4.2)||17.4 (4.37)|
|0.003||-||10.26 (2.65)||16.9 (3.59)||17.1 (3.56)||17.09 (4.2)||17.42 (4.37)|
|0.004||-||-||16.9 (3.6)||17.1 (3.56)||17.1 (4.2)||17.43 (4.37)|
|0.005||-||-||-||17.12 (3.56)||17.11 (4.22)||17.43 (4.38)|
In this section, we describe the training and testing of the proposed RL conservation law solver, and compare it with WENO and some recently proposed supervise learning-based methods for solving conservation laws: L3D  and PINNs [27, 26].
4.1 Training and Testing Setup
In this subsection, we explain the general training and testing setup. We train the RL policy network on the Burgers equation, whose flux is computed as . In all the experiments, we set the left-shift and the right shift . The state function will generate two vectors:
for computing and respectively. and will be passed into the same policy neural network to produce the desired actions, as described in Section 3.2. The reward function simply computes the infinity norm, i.e.,
The policy network is a feed-forward Multi-layer Perceptron with hidden layers, each has 64 neurons and use ReLU  as the activation function. We use the Twin Delayed Deep Deterministic (TD3) policy gradient algorithm  to train the RL policy.
We consider the following three settings of the Burgers Equation:
Inviscid: inviscid Burgers Equation with different random initial conditions:
The reason for designing such kind of initial conditions is to ensure they have similar degree of smoothness and thus similar level of difficulty in learning. For training, we randomly generate such initial conditions, and we always set to ensure enough shocks in the solutions. For testing, we randomly generate another initial conditions in the same form with chosen in . During training, the computation domain is , and the terminal time is randomly sampled from with equal probability; during test, the terminal time is fixed at . We train on two different grid sizes : .
Forcing‡‡ ‡ This setting is exactly the same as the training/testing setting used in L3D .: Viscous Burgers Equation with forcing, and a fixed initial value.
During training, is fixed at , and we test with in . 50 random forcings are generated, the first half is used for training, and the second half is used for testing. During training, the computation domain is , and the terminal time is randomly sampled from with equal probability. The test terminal time is fixed at . We train on two different grid sizes: .
Viscous§§ § Note that the viscous Burgers Equation is easier to solve compared with the inviscid version, as the viscous term smooths out the solution and helps remove shocks in the solution.: Viscous Burgers Equation with different random initial conditions: , and . The viscous term is handled using central difference schemes. The initial conditions are generated in the same form as in equation 4.2. For both training and testing, we sample 25 initial conditions with in . For training, we fix , while for testing, we test with different values of in . The computation domain, and training grid sizes are the same as the inviscid version.
When training the RL agent, we use the Euler scheme for temporal discretization. During test, we always use the 4th order Runge-Kutta (RK4) for temporal discretization. The true solution needed for computing the reward and for evaluation is generated using WENO on the same computation domain with a much finer grid (, ) and use the RK4 as the temporal discretization. In the following, we denote the policy network that generates the weights of the WENO fluxes (as described in Section 3.2) as RL-WENO. In all the settings, we compare RL-WENO with WENO. As L3D is (only) demonstrated to work in the Forcing setting, we also compare RL-WENO with L3D in this setting. We also compare RL-WENO with PINNs in both the inviscid and viscous setting. To demonstrate the ability to perform out-of-distribution generalization of RL-WENO, we train RL-WENO on one of the three settings and test it on the other two settings without fine-tuning or re-training. For all experiments, the evaluation metric is the relative L2 error compared with the true solution computed using WENO on a finer grid as described above. See Table 3 for a summary and road-map for all the experiments.
We also give a summary of the inference time for RL-WENO, WENO, L3D, and PINNs in Table 4. RL-WENO, WENO and L3D are tested in the forcing setting, averaged over 25 random forcings, and PINNs is tested in the viscous setting, averaged over 25 different random initial conditions. For PINNs, as it needs to be trained on each different initial condition, the inference time is the same as the training time.
|(a) Smooth regions,||(b) Near singularities,|
4.2 inviscid Burgers Equation with random initial conditions
Generalization to new mesh sizes, flux function, and temporal discretization.
We first test whether the RL policy trained in this setting can generalize to different temporal discretization schemes, mesh sizes and flux functions that are not included in training. Table 1 and Table 2 present the comparison results, where the number shows the relative error (computed as with the 2-norm taking over all ) between the approximated solution and the true solution , averaged over all evolving steps () and 25 random test initial values. Numbers in the bracket shows the standard deviation over the 25 initial values. Several entries in the table are marked as ‘-’ because the corresponding CFL number is not small enough to guarantee convergence.
Our experimental results show that, compared with the high order accurate WENO (5th order accurate in space and 4th order accurate in time), the linear weights learned by RL not only achieves smaller errors, but also generalizes well to:
New time discretization scheme: trained with Euler, tested with RK4;
Figure 1 shows some examples of the solutions. As one can see, the solutions generated by RL-WENO not only achieve the same accuracy as WENO at smooth regions, but also have clear advantage over WENO near singularities which is particularly challenging for numerical PDE solvers and important in applications.
More analysis on how RL-WENO chooses the weights.
First, as shown in Figure 2, RL-WENO can indeed correctly determine upwind directions and generate local numerical schemes in an adaptive fashion. More interestingly, Figure 2 further shows that comparing to WENO, RL-WENO seems to be able to select stencils in a different way from it, and eventually leads to a more accurate solution.
As mentioned in Section 2.2, WENO itself already achieves an optimal order of accuracy in the smooth regions. Since RL-WENO can further improve upon WENO, it should have achieved higher accuracy especially near singularities. Therefore, we provide additional demonstrations on how RL-WENO performs in the smooth/singular regions. We run RL-WENO and WENO on a set of initial conditions, and record the approximation errors at every spatial-temporal location. We then separate the errors in the smooth and singular regions, and compute the distribution of the errors on the entire spatial-temporal grids with multiple initial conditions. The results are shown in figure 3. In figure 3, the -axis is the logarithmic (base 10) value of the error and the y-axis is the number of grid points whose error is less than the corresponding value on the -axis, i.e., the accumulated distribution of the errors. The results illustrate that RL-WENO indeed performs better than WENO near singularities.
Given that RL-WENO performs better than WENO at singularities, we further investigate how it inferences the weights differently from WENO. Figure 4 shows the distribution of weights computed by both methods near singularities. Again, it can be seen that RL-WENO generates the weights in a different way from WENO. Interestingly, the smaller standard deviation of RL-WENO also seems to show that RL-WENO is more confident on its choice of the weights. These results suggest that the proposed RL framework has the potential to surpass human experts in designing numerical schemes for conservation laws.
Comparison with PINNs
|0||0.02||10.81 (3.96)||11.13 (3.83)||40.06(18.82)|
|0.04||18.89 (12.69)||19.48 (9.33)||51.61(23.68)|
|0.05||28.48 (33.04)||35.93 (27.61)||51.18(18.31)|
|0.01||0.02||1.64 (0.57)||1.94 (0.55)||34.09(26.92)|
|0.04||4.29 (0.93)||4.93 (0.73)||48.28(29.71)|
|0.05||7.6 (3.94)||7.66 (1.71)||47.95(19.86)|
|0.02||0.02||0.74 (0.35)||0.87 (0.37)||31.13(34.35)|
|0.04||1.91 (0.58)||2.33 (0.51)||47.41(35.84)|
|0.05||2.97 (1.07)||3.64 (0.77)||43.05(20.1))|
|0.04||0.02||0.42 (0.21)||0.41 (0.21)||34.98(37.83)|
|0.04||0.87 (0.38)||0.96 (0.36)||52.27(44.74)|
|0.05||1.38 (0.61)||1.52 (0.5)||47.0(24.47)|
PINNs  directly uses a deep neural network to approximate the solution to the conservation law (2.1). Therefore, it needs to be trained individually for each different initial condition, as the underlying solution is different. The implementation of PINNs is based on the released codes by the authors ¶¶ ¶ https://github.com/maziarraissi/PINNs. Note that we assume periodic boundary condition of (2.1). Therefore, we modified the released codes of PINNs according to another related online codes ∥∥ ∥ https://github.com/MinglangYin/PyTorchTutorial/tree/master/PINNs. The comparison result is shown in Table 5. In fact, PINNs performs poorly with relatively complicated initial conditions such as the ones we chose in Eq. (4.2). Indeed, when we simplify the initial condition, e.g., , the performance of PINNs is significantly improved (see a comparison in Figure 5). To resolve this issue, we may choose more expressive neural networks to approximate the solutions. However, this may also increase the difficulty of training.
Out-of-distribution generalization to the other two test settings.
We now test if RL-WENO trained in the inviscid setting can generalize to the other two settings: viscous and forcing. Table 6 and 7 show the results. Surprisingly, RL-WENO generalizes relatively well to these two test settings without any re-training or fine-tuning. We attribute the good generalization ability of RL-WENO to our careful action design, which essentially makes RL-WENO a meta-learner under the WENO framework and thus have strong out-of-distribution generalization.
|0.02||4.75 (0.73)||4.74 (0.7)||2.38 (0.46)||2.59 (0.45)||1.07 (0.2)||1.02 (0.27)|
|0.04||10.78 (1.4)||10.3 (1.35)||6.95 (1.15)||6.6 (0.95)||3.84 (0.7)||3.68 (0.62)|
|0.05||13.97 (1.93)||13.55 (2.07)||9.76 (1.44)||9.33 (1.4)||5.67 (0.89)||5.42 (0.81)|
|0.02||1.85 (0.58)||1.94 (0.55)||0.85 (0.38)||0.87 (0.37)||0.45 (0.22)||0.41 (0.21)|
|0.04||4.91 (1.64)||4.93 (0.73)||2.34 (0.82)||2.33 (0.51)||1.14 (0.58)||0.96 (0.36)|
|0.05||8.31 (5.49)||7.66 (1.71)||3.59 (1.05)||3.64 (0.77)||1.84 (0.88)||1.52 (0.5)|
4.3 Viscous Burgers Equation with forcing and fixed initial value
|0.01||0.02||4.64 (0.7)||4.74 (0.7)||3.09 (0.9)|
|0.04||10.36 (1.45)||10.3 (1.35)||nan (nan)|
|0.05||13.33 (2.05)||13.55 (2.07)||nan (nan)|
|0.02||0.02||2.41 (0.43)||2.59 (0.45)||6.89 (0.74)|
|0.04||6.53 (1.13)||6.6 (0.95)||4.08 (1.12)|
|0.05||9.19 (1.54)||9.33 (1.4)||7.58 (1.11)|
|0.04||0.02||1.09 (0.21)||1.02 (0.27)||11.01 (1.3)|
|0.04||3.42 (0.65)||3.68 (0.62)||11.22 (1.22)|
|0.05||4.97 (0.91)||5.42 (0.81)||9.37 (1.21)|
Comparision between WENO and L3D.
In this section, we train RL-WENO in the forcing setting, and compare it with WENO and a recently proposed supervise learning-based method: L3D . Note that this setting we use (Eq. (4.3)) is exactly the same as the training/testing setting used in L3D.
For L3D, we use the code released by the authors**** ** https://github.com/google/data-driven-discretization-1d, and train a model using their provided dataset. After training, we test the trained model on different grid sizes and viscosity.
The test results are shown in Table 8. We observe that in three cases, L3D reports lower error than WENO and RL-WENO, and in all other test cases, L3D reports higher error than RL-WENO and WENO. Notably, the trained model by L3D diverges in two of the test cases, where the solution blows up during the evolution of the equation. This shows that L3D has worse generalization compared with RL-WENO.
Out-of-distribution generalization to the other two settings.
|0.02||13.06 (3.22)||11.13 (3.83)||1.9 (0.58)||1.94 (0.55)||0.92 (0.37)||0.87 (0.37)||0.49 (0.22)||0.41 (0.21)|
|0.04||18.7 (7.65)||19.48 (9.33)||4.76 (1.19)||4.93 (0.73)||2.3 (0.78)||2.33 (0.51)||1.13 (0.62)||0.96 (0.36)|
|0.05||28.57 (22.3)||35.88 (27.48)||7.75 (2.76)||7.66 (1.71)||3.51 (1.24)||3.64 (0.77)||1.72 (0.95)||1.52 (0.5)|
The generalization results of RL-WENO to the inviscid and viscous settings are shown in Table 9. We again see that RL-WENO generalizes relatively well. We tried to train L3D in the inviscid/viscous settings, but could not get any decent models. Besides, the L3D model trained under the forcing setting cannot generalize to other settings (the model blows up during the evolution of the equation). In contrast, RL-WENO has much stronger flexibility and generalization: decent policies can be obtained when trained under various settings, and the model trained under one setting can generalize relatively well to other settings. Table 3 gives a thorough discussion of the generalization ability of L3D and RL-WENO.
4.4 Viscous Burgers Equation with random initial conditions
|0.02||11.17 (3.8)||11.13 (3.83)||1.64 (0.57)||1.94 (0.55)||0.74 (0.35)||0.87 (0.37)||0.42 (0.21)||0.41 (0.21)|
|0.04||19.37 (11.67)||19.48 (9.33)||4.29 (0.93)||4.93 (0.73)||1.91 (0.58)||2.33 (0.51)||0.87 (0.38)||0.96 (0.36)|
|0.05||41.44 (45.05)||35.88 (27.48)||7.6 (3.94)||7.66 (1.71)||2.97 (1.07)||3.64 (0.77)||1.38 (0.61)||1.52 (0.5)|
Comparison between WENO and PINNs.
The test results of RL-WENO compared with WENO are shown in the last 3 columns of Table 10. We again see that in most cases, RL-WENO outperforms WENO.
The comparison results of RL-WENO versus PINNs are shown in Table 5. Again, we see that PINNs performs poorly, due to the complexity of our chosen initial values.
Out-of-distribution generalization to the other two test settings.
|0.01||4.66 (0.72)||4.74 (0.7)||2.33 (0.47)||2.59 (0.45)||1.01 (0.19)||1.02 (0.27)|
|0.02||10.28 (1.33)||10.3 (1.35)||6.43 (0.88)||6.6 (0.95)||3.43 (0.61)||3.68 (0.62)|
|0.04||13.38 (2.3)||13.55 (2.07)||9.08 (1.57)||9.33 (1.4)||5.19 (0.92)||5.42 (0.81)|
5 Conclusion and Future Work
In this paper, we proposed a general framework to learn how to solve 1-dimensional conservation laws via deep reinforcement learning. We first discussed how the procedure of numerically solving conservation laws can be naturally cast in the form of Markov Decision Process. We then elaborated how to relate notions in numerical schemes of PDEs with those of reinforcement learning. In particular, we introduced a numerical flux policy which was able to decide on how numerical flux should be designed locally based on the current state of the solution. We carefully design the action of our RL policy to make it a meta-learner. Our numerical experiments showed that the proposed RL based solver was able to outperform high order WENO, which is one of the most delicate and successful numerical solvers for conservation laws. Furthermore, RL based solver has notably better generalization than the supervised learning based approach L3D and PINNs, especially for out-of distribution generalization, i.e. when the test settings are vastly different from the training settings.
As part of the future works, we would like to consider using the numerical flux policy to inference more complicated numerical fluxes with guaranteed consistency and stability. Furthermore, we can use the proposed framework to learn a policy that can generate adaptive grids and the associated numerical schemes. Lastly, we would like consider system of conservation laws in 2nd and 3rd dimensional space.
Bin Dong is supported in part by National Natural Science Foundation of China (NSFC) grant No. 11831002, Beijing Natural Science Foundation (No. 180001) and Beijing Academy of Artificial Intelligence (BAAI).
-  (2016) Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pp. 3981–3989. Cited by: 5th item.
-  (2018) Data-driven discretization: a method for systematic coarse graining of partial differential equations. arXiv preprint arXiv:1808.04930. Cited by: §1.1.
-  (2019) Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences 116 (31), pp. 15344–15349. Cited by: §1.2, §4.3, §4, footnote ‡.
-  (2017) Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations. Journal of Nonlinear Science, pp. 1–57. Cited by: §1.1.
-  (1992) On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, pp. 6–8. Cited by: 5th item.
-  (2019) DeepMoD: deep learning for model discovery in noisy data. arXiv preprint arXiv:1904.09406. Cited by: §1.1.
-  (2020) Controlling oscillations in high-order discontinuous galerkin schemes using artificial viscosity tuned by neural networks. Journal of Computational Physics, pp. 109304. Cited by: §1.1.
-  (2018) Decouple learning for parameterized image operators. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 442–458. Cited by: 5th item.
-  (2018) A multiscale neural network based on hierarchical matrices. arXiv preprint arXiv:1807.01883. Cited by: §1.1.
-  (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1126–1135. Cited by: 5th item.
-  (2018) Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477. Cited by: §4.1.
-  (2016) Deep learning. MIT press. Cited by: §4.1.
-  (2018) Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences 115 (34), pp. 8505–8510. Cited by: §1.1.
-  (2010) A new tvd flux-limiter method for solving nonlinear hyperbolic equations. Journal of Computational and Applied Mathematics 234 (5), pp. 1395–1403. Cited by: §1.
-  (2017) Noise-blind image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3510–3518. Cited by: 5th item.
-  (2017) Solving parametric pde problems with artificial neural networks. arXiv preprint arXiv:1707.03351. Cited by: §1.1.
-  (2018) SwitchNet: a neural network model for forward and inverse scattering problems. arXiv preprint arXiv:1810.09675. Cited by: §1.1.
-  (1992) Numerical methods for conservation laws. Vol. 132, Springer. Cited by: §1, §2.2.
-  (2002) Finite volume methods for hyperbolic problems. Vol. 31, Cambridge university press. Cited by: §1.
-  (2016) Learning to optimize. arXiv preprint arXiv:1606.01885. Cited by: 5th item.
-  (2019) Variational training of neural network approximations of solution maps for physical models. arXiv preprint arXiv:1905.02789. Cited by: §1.1.
-  (1994) Weighted essentially non-oscillatory schemes. Journal of computational physics 115 (1), pp. 200–212. Cited by: §1, §2.2.
-  (2019) Constraint-aware neural networks for riemann problems. arXiv preprint arXiv:1904.12794. Cited by: §1.1.
-  (1993) Matching pursuits with time-frequency dictionaries. IEEE Transactions on signal processing 41 (12), pp. 3397–3415. Cited by: 6th item.
-  (2019) Solving irregular and data-enriched differential equations using deep neural networks. arXiv preprint arXiv:1905.04351. Cited by: §1.1.
-  (2019) Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics 378, pp. 686–707. Cited by: §1.1, §1.2, §4.
-  (2017) Physics informed deep learning (part i): data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10561. Cited by: §1.1, §1.2, §4.2, §4.
-  (2018) An artificial neural network as a troubled-cell indicator. Journal of Computational Physics 367, pp. 166–191. Cited by: §1.1.
-  (1987) Evolutionary principles in self-referential learning. On learning how to learn: The meta-meta-… hook.) Diploma thesis, Institut f. Informatik, Tech. Univ. Munich. Cited by: 5th item.
-  (1998) Essentially non-oscillatory and weighted essentially non-oscillatory schemes for hyperbolic conservation laws. In Advanced numerical approximation of nonlinear hyperbolic equations, pp. 325–432. Cited by: 3rd item, §1, §2.2.
-  (2018) DGM: a deep learning algorithm for solving partial differential equations. Journal of computational physics 375, pp. 1339–1364. Cited by: §1.1.
-  (2018) Meta-learning: a survey. arXiv preprint arXiv:1810.03548. Cited by: 5th item.
-  (2019) General solutions for nonlinear differential equations: a rule-based self-learning approach using deep reinforcement learning. Computational Mechanics, pp. 1–14. Cited by: §1.1.
-  (2018) The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics 6 (1), pp. 1–12. Cited by: §1.1.
-  (2019) Dynamically unfolding recurrent restorer: a moving endpoint control method for image restoration. ICLR. Cited by: 5th item.
-  (2005) A fast sweeping method for eikonal equations. Mathematics of computation 74 (250), pp. 603–627. Cited by: 6th item.