Abstract
Language drift has been one of the major obstacles to train language modelsthrough interaction. When word-based conversational agents are trained towardscompleting a task, they tend to invent their language rather than leveragingnatural language. In recent literature, two general methods partially counterthis phenomenon: Supervised Selfplay (S2P) and Seeded Iterated Learning (SIL).While S2P jointly trains interactive and supervised losses to counter thedrift, SIL changes the training dynamics to prevent language drift fromoccurring. In this paper, we first highlight their respective weaknesses, i.e.,late-stage training collapses and higher negative likelihood when evaluated onhuman corpus. Given these observations, we introduce Supervised Seeded IteratedLearning to combine both methods to minimize their respective weaknesses. Wethen show the effectiveness of \algo in the language-drift translation game.