Abstract
Standard reinforcement learning (RL) algorithms train agents to maximizegiven reward functions. However, many real-world applications of RL requireagents to also satisfy certain constraints which may, for example, be motivatedby safety concerns. Constrained RL algorithms approach this problem by trainingagents to maximize given reward functions while respecting \textit{explicitly}defined constraints. However, in many cases, manually designing accurateconstraints is a challenging task. In this work, given a reward function and aset of demonstrations from an expert that maximizes this reward function whilerespecting \textit{unknown} constraints, we propose a framework to learn themost likely constraints that the expert respects. We then train agents tomaximize the given reward function subject to the learned constraints. Previousworks in this regard have either mainly been restricted to tabular settings orspecific types of constraints or assume knowledge of transition dynamics of theenvironment. In contrast, we empirically show that our framework is able tolearn arbitrary \textit{Markovian} constraints in high-dimensions in amodel-free setting.