Abstract
Human-in-the-loop Reinforcement Learning (HRL) aims to integrate humanguidance with Reinforcement Learning (RL) algorithms to improve sampleefficiency and performance. A common type of human guidance in HRL is binaryevaluative "good" or "bad" feedback for queried states and actions. However,this type of learning scheme suffers from the problems of weak supervision andpoor efficiency in leveraging human feedback. To address this, we presentEXPAND (EXPlanation AugmeNted feeDback) which provides a visual explanation inthe form of saliency maps from humans in addition to the binary feedback.EXPAND employs a state perturbation approach based on salient information inthe state to augment the binary feedback. We choose five tasks, namelyPixel-Taxi and four Atari games, to evaluate this approach. We demonstrate theeffectiveness of our method using two metrics: environment sample efficiencyand human feedback sample efficiency. We show that our method significantlyoutperforms previous methods. We also analyze the results qualitatively byvisualizing the agent's attention. Finally, we present an ablation study toconfirm our hypothesis that augmenting binary feedback with state salientinformation results in a boost in performance.