Abstract
The standard formulation of Reinforcement Learning lacks a practical way ofspecifying what are admissible and forbidden behaviors. Most often,practitioners go about the task of behavior specification by manuallyengineering the reward function, a counter-intuitive process that requiresseveral iterations and is prone to reward hacking by the agent. In this work,we argue that constrained RL, which has almost exclusively been used for safeRL, also has the potential to significantly reduce the amount of work spent forreward specification in applied RL projects. To this end, we propose to specifybehavioral preferences in the CMDP framework and to use Lagrangian methods toautomatically weigh each of these behavioral constraints. Specifically, weinvestigate how CMDPs can be adapted to solve goal-based tasks while adheringto several constraints simultaneously. We evaluate this framework on a set ofcontinuous control tasks relevant to the application of Reinforcement Learningfor NPC design in video games.