Due to the capability of deep learning to perform well in high dimensionalproblems, deep reinforcement learning agents perform well in challenging taskssuch as Atari 2600 games. However, clearly explaining why a certain action istaken by the agent can be as important as the decision itself. Deepreinforcement learning models, as other deep learning models, tend to be opaquein their decision-making process. In this work, we propose to make deepreinforcement learning more transparent by visualizing the evidence on whichthe agent bases its decision. In this work, we emphasize the importance ofproducing a justification for an observed action, which could be applied to ablack-box decision agent.