Abstract
Self-play, characterized by agents' interactions with copies or past versionsof themselves, has recently gained prominence in reinforcement learning (RL).This paper first clarifies the preliminaries of self-play, including themulti-agent reinforcement learning framework and basic game theory concepts.Then, it provides a unified framework and classifies existing self-playalgorithms within this framework. Moreover, the paper bridges the gap betweenthe algorithms and their practical implications by illustrating the role ofself-play in different scenarios. Finally, the survey highlights openchallenges and future research directions in self-play. This paper is anessential guide map for understanding the multifaceted landscape of self-playin RL.