Reinforcement Learning (RL) algorithms can in principle acquire complexrobotic skills by learning from large amounts of data in the real world,collected via trial and error. However, most RL algorithms use a carefullyengineered setup in order to collect data, requiring human supervision andintervention to provide episodic resets. This is particularly evident inchallenging robotics problems, such as dexterous manipulation. To make datacollection scalable, such applications require reset-free algorithms that areable to learn autonomously, without explicit instrumentation or humanintervention. Most prior work in this area handles single-task learning.However, we might also want robots that can perform large repertoires ofskills. At first, this would appear to only make the problem harder. However,the key observation we make in this work is that an appropriately chosenmulti-task RL setting actually alleviates the reset-free learning challenge,with minimal additional machinery required. In effect, solving a multi-taskproblem can directly solve the reset-free problem since different combinationsof tasks can serve to perform resets for other tasks. By learning multipletasks together and appropriately sequencing them, we can effectively learn allof the tasks together reset-free. This type of multi-task learning caneffectively scale reset-free learning schemes to much more complex problems, aswe demonstrate in our experiments. We propose a simple scheme for multi-tasklearning that tackles the reset-free learning problem, and show itseffectiveness at learning to solve complex dexterous manipulation tasks in bothhardware and simulation without any explicit resets. This work shows theability to learn dexterous manipulation behaviors in the real world with RLwithout any human intervention.