Abstract
When a person is not satisfied with how a robot performs a task, they canintervene to correct it. Reward learning methods enable the robot to adapt itsreward function online based on such human input, but they rely on handcraftedfeatures. When the correction cannot be explained by these features, recentwork in deep Inverse Reinforcement Learning (IRL) suggests that the robot couldask for task demonstrations and recover a reward defined over the raw statespace. Our insight is that rather than implicitly learning about the missingfeature(s) from demonstrations, the robot should instead ask for data thatexplicitly teaches it about what it is missing. We introduce a new type ofhuman input in which the person guides the robot from states where the featurebeing taught is highly expressed to states where it is not. We propose analgorithm for learning the feature from the raw state space and integrating itinto the reward function. By focusing the human input on the missing feature,our method decreases sample complexity and improves generalization of thelearned reward over the above deep IRL baseline. We show this in experimentswith a physical 7DOF robot manipulator, as well as in a user study conducted ina simulated environment.