Abstract
The success of recent Artificial Intelligence (AI) models has beenaccompanied by the opacity of their internal mechanisms, due notably to the useof deep neural networks. In order to understand these internal mechanisms andexplain the output of these AI models, a set of methods have been proposed,grouped under the domain of eXplainable AI (XAI). This paper focuses on asub-domain of XAI, called eXplainable Reinforcement Learning (XRL), which aimsto explain the actions of an agent that has learned by reinforcement learning.We propose an intuitive taxonomy based on two questions "What" and "How". Thefirst question focuses on the target that the method explains, while the secondrelates to the way the explanation is provided. We use this taxonomy to providea state-of-the-art review of over 250 papers. In addition, we present a set ofdomains close to XRL, which we believe should get attention from the community.Finally, we identify some needs for the field of XRL.