Abstract
For an autonomous agent to fulfill a wide range of user-specified goals attest time, it must be able to learn broadly applicable and general-purposeskill repertoires. Furthermore, to provide the requisite level of generality,these skills must handle raw sensory input such as images. In this paper, wepropose an algorithm that acquires such general-purpose skills by combiningunsupervised representation learning and reinforcement learning ofgoal-conditioned policies. Since the particular goals that might be required attest-time are not known in advance, the agent performs a self-supervised"practice" phase where it imagines goals and attempts to achieve them. We learna visual representation with three distinct purposes: sampling goals forself-supervised practice, providing a structured transformation of raw sensoryinputs, and computing a reward signal for goal reaching. We also propose aretroactive goal relabeling scheme to further improve the sample-efficiency ofour method. Our off-policy algorithm is efficient enough to learn policies thatoperate on raw image observations and goals for a real-world robotic system,and substantially outperforms prior techniques.