Abstract
For autonomous agents to successfully integrate into human-centeredenvironments, agents should be able to learn from and adapt to humans in theirnative settings. Preference-based reinforcement learning (PbRL) is a promisingapproach that learns reward functions from human preferences. This enables RLagents to adapt their behavior based on human desires. However, humans live ina world full of diverse information, most of which is not relevant tocompleting a particular task. It becomes essential that agents learn to focuson the subset of task-relevant environment features. Unfortunately, prior workhas largely ignored this aspect; primarily focusing on improving PbRLalgorithms in standard RL environments that are carefully constructed tocontain only task-relevant features. This can result in algorithms that may noteffectively transfer to a more noisy real-world setting. To that end, this workproposes R2N (Robust-to-Noise), the first PbRL algorithm that leveragesprinciples of dynamic sparse training to learn robust reward models that canfocus on task-relevant features. We study the effectiveness of R2N in theExtremely Noisy Environment setting, an RL problem setting where up to 95% ofthe state features are irrelevant distractions. In experiments with a simulatedteacher, we demonstrate that R2N can adapt the sparse connectivity of itsneural networks to focus on task-relevant features, enabling R2N tosignificantly outperform several state-of-the-art PbRL algorithms in multiplelocomotion and control environments.