Abstract
Reinforcement Learning (RL) is a popular machine learning paradigm whereintelligent agents interact with the environment to fulfill a long-term goal.Driven by the resurgence of deep learning, Deep RL (DRL) has witnessed greatsuccess over a wide spectrum of complex control tasks. Despite the encouragingresults achieved, the deep neural network-based backbone is widely deemed as ablack box that impedes practitioners to trust and employ trained agents inrealistic scenarios where high security and reliability are essential. Toalleviate this issue, a large volume of literature devoted to shedding light onthe inner workings of the intelligent agents has been proposed, by constructingintrinsic interpretability or post-hoc explainability. In this survey, weprovide a comprehensive review of existing works on eXplainable RL (XRL) andintroduce a new taxonomy where prior works are clearly categorized intomodel-explaining, reward-explaining, state-explaining, and task-explainingmethods. We also review and highlight RL methods that conversely leverage humanknowledge to promote learning efficiency and final performance of agents whilethis kind of method is often ignored in XRL field. Some open challenges andopportunities in XRL are discussed. This survey intends to provide a high-levelsummarization and better understanding of XRL and to motivate future researchon more effective XRL solutions. Corresponding open source codes are collectedand categorized athttps://github.com/Plankson/awesome-explainable-reinforcement-learning.