Abstract
Bilevel reinforcement learning (RL), which features intertwined two-levelproblems, has attracted growing interest recently. The inherent non-convexityof the lower-level RL problem is, however, to be an impediment to developingbilevel optimization methods. By employing the fixed point equation associatedwith the regularized RL, we characterize the hyper-gradient via fullyfirst-order information, thus circumventing the assumption of lower-levelconvexity. This, remarkably, distinguishes our development of hyper-gradientfrom the general AID-based bilevel frameworks since we take advantage of thespecific structure of RL problems. Moreover, we design both model-based andmodel-free bilevel reinforcement learning algorithms, facilitated by access tothe fully first-order hyper-gradient. Both algorithms enjoy the convergencerate $O(\epsilon^{-1})$. To extend the applicability, a stochastic version ofthe model-free algorithm is proposed, along with results on its iteration andsample complexity. In addition, numerical experiments demonstrate that thehyper-gradient indeed serves as an integration of exploitation and exploration.