Abstract
Reinforcement learning (RL) algorithms allow agents to learn skills andstrategies to perform complex tasks without detailed instructions or expensivelabelled training examples. That is, RL agents can learn, as we learn. Giventhe importance of learning in our intelligence, RL has been thought to be oneof key components to general artificial intelligence, and recent breakthroughsin deep reinforcement learning suggest that neural networks (NN) are naturalplatforms for RL agents. However, despite the efficiency and versatility ofNN-based RL agents, their decision-making remains incomprehensible, reducingtheir utilities. To deploy RL into a wider range of applications, it isimperative to develop explainable NN-based RL agents. Here, we propose a methodto derive a secondary comprehensible agent from a NN-based RL agent, whosedecision-makings are based on simple rules. Our empirical evaluation of thissecondary agent's performance supports the possibility of building acomprehensible and transparent agent using a NN-based RL agent.