Abstract
This work uses the entropy-regularised relaxed stochastic control perspectiveas a principled framework for designing reinforcement learning (RL) algorithms.Herein agent interacts with the environment by generating noisy controlsdistributed according to the optimal relaxed policy. The noisy policies, on theone hand, explore the space and hence facilitate learning but, on the otherhand, introduce bias by assigning a positive probability to non-optimalactions. This exploration-exploitation trade-off is determined by the strengthof entropy regularisation. We study algorithms resulting from two entropyregularisation formulations: the exploratory control approach, where entropy isadded to the cost objective, and the proximal policy update approach, whereentropy penalises the divergence of policies between two consecutive episodes.We analyse the finite horizon continuous-time linear-quadratic (LQ) RL problemfor which both algorithms yield a Gaussian relaxed policy. We quantify theprecise difference between the value functions of a Gaussian policy and itsnoisy evaluation and show that the execution noise must be independent acrosstime. By tuning the frequency of sampling from relaxed policies and theparameter governing the strength of entropy regularisation, we prove that theregret, for both learning algorithms, is of the order $\mathcal{O}(\sqrt{N}) $(up to a logarithmic factor) over $N$ episodes, matching the best known resultfrom the literature.