Abstract
In this work, we devise a new, general-purpose reinforcement learningstrategy for the optimal control of parametric partial differential equations(PDEs). Such problems frequently arise in applied sciences and engineering andentail a significant complexity when control and/or state variables aredistributed in high-dimensional space or depend on varying parameters.Traditional numerical methods, relying on either iterative minimizationalgorithms or dynamic programming, while reliable, often become computationallyinfeasible. Indeed, in either way, the optimal control problem must be solvedfor each instance of the parameters, and this is out of reach when dealing withhigh-dimensional time-dependent and parametric PDEs. In this paper, we proposeHypeRL, a deep reinforcement learning (DRL) framework to overcome thelimitations shown by traditional methods. HypeRL aims at approximating theoptimal control policy directly. Specifically, we employ an actor-critic DRLapproach to learn an optimal feedback control strategy that can generalizeacross the range of variation of the parameters. To effectively learn suchoptimal control laws, encoding the parameter information into the DRL policyand value function neural networks (NNs) is essential. To do so, HypeRL usestwo additional NNs, often called hypernetworks, to learn the weights and biasesof the value function and the policy NNs. We validate the proposed approach ontwo PDE-constrained optimal control benchmarks, namely a 1DKuramoto-Sivashinsky equation and a 2D Navier-Stokes equations, by showing thatthe knowledge of the PDE parameters and how this information is encoded, i.e.,via a hypernetwork, is an essential ingredient for learning parameter-dependentcontrol policies that can generalize effectively to unseen scenarios and forimproving the sample efficiency of such policies.