Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL

  • 2022-01-26 14:03:48
  • Simon Hirlaender, Niky Bruchon
  • 0

Abstract

Reinforcement learning holds tremendous promise in accelerator controls. Theprimary goal of this paper is to show how this approach can be utilised on anoperational level on accelerator physics problems. Despite the success ofmodel-free reinforcement learning in several domains, sample-efficiency stillis a bottle-neck, which might be encompassed by model-based methods. We comparewell-suited purely model-based to model-free reinforcement learning applied tothe intensity optimisation on the FERMI FEL system. We find that themodel-based approach demonstrates higher representational power andsample-efficiency, while the asymptotic performance of the model-free method isslightly superior. The model-based algorithm is implemented in a DYNA-styleusing an uncertainty aware model, and the model-free algorithm is based ontailored deep Q-learning. In both cases, the algorithms were implemented in away, which presents increased noise robustness as omnipresent in acceleratorcontrol problems. Code is released inhttps://github.com/MathPhysSim/FERMI_RL_Paper.

 

Quick Read (beta)

loading the full paper ...