Abstract
The classic Reinforcement Learning (RL) formulation concerns the maximizationof a scalar reward function. More recently, convex RL has been introduced toextend the RL formulation to all the objectives that are convex functions ofthe state distribution induced by a policy. Notably, convex RL covers severalrelevant applications that do not fall into the scalar formulation, includingimitation learning, risk-averse RL, and pure exploration. In classic RL, it iscommon to optimize an infinite trials objective, which accounts for the statedistribution instead of the empirical state visitation frequencies, even thoughthe actual number of trajectories is always finite in practice. This istheoretically sound since the infinite trials and finite trials objectives canbe proved to coincide and thus lead to the same optimal policy. In this paper,we show that this hidden assumption does not hold in the convex RL setting. Inparticular, we show that erroneously optimizing the infinite trials objectivein place of the actual finite trials one, as it is usually done, can lead to asignificant approximation error. Since the finite trials setting is the defaultin both simulated and real-world RL, we believe shedding light on this issuewill lead to better approaches and methodologies for convex RL, impactingrelevant research areas such as imitation learning, risk-averse RL, and pureexploration among others.