Abstract
Many currently deployed Reinforcement Learning agents work in an environmentshared with humans, be them co-workers, users or clients. It is desirable thatthese agents adjust to people's preferences, learn faster thanks to their help,and act safely around them. We argue that most current approaches that learnfrom human feedback are unsafe: rewarding or punishing the agent a-posterioricannot immediately prevent it from wrong-doing. In this paper, we extend PolicyGradient to make it robust to external directives, that would otherwise breakthe fundamentally on-policy nature of Policy Gradient. Our technique, DirectedPolicy Gradient (DPG), allows a teacher or backup policy to override the agentbefore it acts undesirably, while allowing the agent to leverage human adviceor directives to learn faster. Our experiments demonstrate that DPG makes theagent learn much faster than reward-based approaches, while requiring an orderof magnitude less advice.