Abstract
In reinforcement learning, it is common to let an agent interact for a fixedamount of time with its environment before resetting it and repeating theprocess in a series of episodes. The task that the agent has to learn caneither be to maximize its performance over (i) that fixed period, or (ii) anindefinite period where time limits are only used during training to diversifyexperience. In this paper, we provide a formal account for how time limitscould effectively be handled in each of the two cases and explain why not doingso can cause state-aliasing and invalidation of experience replay, leading tosuboptimal policies and training instability. In case (i), we argue that theterminations due to time limits are in fact part of the environment, and thus anotion of the remaining time should be included as part of the agent's input toavoid violation of the Markov property. In case (ii), the time limits are notpart of the environment and are only used to facilitate learning. We argue thatthis insight should be incorporated by bootstrapping from the value of thestate at the end of each partial episode. For both cases, we illustrateempirically the significance of our considerations in improving the performanceand stability of existing reinforcement learning algorithms, showingstate-of-the-art results on several control tasks.