Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control

  • 2018-06-13 03:47:12
  • Yangchen Pan, Amir-massoud Farahmand, Martha White, Saleh Nabi, Piyush Grover, Daniel Nikovski
  • 2

Abstract

Recent work has shown that reinforcement learning (RL) is a promisingapproach to control dynamical systems described by partial differentialequations (PDE). This paper shows how to use RL to tackle more general PDEcontrol problems that have continuous high-dimensional action spaces withspatial relationship among action dimensions. In particular, we propose theconcept of action descriptors, which encode regularities amongspatially-extended action dimensions and enable the agent to controlhigh-dimensional action PDEs. We provide theoretical evidence suggesting thatthis approach can be more sample efficient compared to a conventional approachthat treats each action dimension separately and does not explicitly exploitthe spatial regularity of the action space. The action descriptor approach isthen used within the deep deterministic policy gradient algorithm. Experimentson two PDE control problems, with up to 256-dimensional continuous actions,show the advantage of the proposed approach over the conventional one.

 

Quick Read (beta)

loading the full paper ...