Abstract
Despite the significant advancements in Deep Reinforcement Learning (RL)observed in the last decade, the amount of training experience necessary tolearn effective policies remains one of the primary concerns both in simulatedand real environments. Looking to solve this issue, previous work has shownthat improved training efficiency can be achieved by separately modeling agentand environment, but usually requiring a supervisory agent mask. In contrast to RL, humans can perfect a new skill from a small number oftrials and in most cases do so without a supervisory signal, makingneuroscientific studies of human development a valuable source of inspirationfor RL. In particular, we explore the idea of motor prediction, which statesthat humans develop an internal model of themselves and of the consequencesthat their motor commands have on the immediate sensory inputs. Our insight isthat the movement of the agent provides a cue that allows the duality betweenagent and environment to be learned. To instantiate this idea, we present Ego-Foresight, a self-supervised methodfor disentangling agent and environment based on motion and prediction. Ourmain finding is self-supervised agent-awareness by visuomotor prediction of theagent improves sample-efficiency and performance of the underlying RLalgorithm. To test our approach, we first study its ability to visually predict agentmovement irrespective of the environment, in simulated and real-world roboticdata. Then, we integrate Ego-Foresight with a model-free RL algorithm to solvesimulated robotic tasks, showing that self-supervised agent-awareness canimprove sample-efficiency and performance in RL.