Reinforcement learning is a powerful approach to learn behaviour throughinteractions with an environment. However, behaviours are usually learned in apurely reactive fashion, where an appropriate action is selected based on anobservation. In this form, it is challenging to learn when it is necessary toexecute new decisions. This makes learning inefficient, especially inenvironments that need various degrees of fine and coarse control. To addressthis, we propose a proactive setting in which the agent not only selects anaction in a state but also for how long to commit to that action. Our TempoRLapproach introduces skip connections between states and learns a skip-policyfor repeating the same action along these skips. We demonstrate theeffectiveness of TempoRL on a variety of traditional and deep RL environments,showing that our approach is capable of learning successful policies up to anorder of magnitude faster than vanilla Q-learning.