Efficient exploration remains a challenging research problem in reinforcementlearning, especially when an environment contains large state spaces, deceptivelocal optima, or sparse rewards. To tackle this problem, we present adiversity-driven approach for exploration, which can be easily combined withboth off- and on-policy reinforcement learning algorithms. We show that bysimply adding a distance measure to the loss function, the proposed methodologysignificantly enhances an agent's exploratory behaviors, and thus preventingthe policy from being trapped in local optima. We further propose an adaptivescaling method for stabilizing the learning process. Our experimental resultsin Atari 2600 show that our method outperforms baseline approaches in severaltasks in terms of mean scores and exploration efficiency.