Abstract
Training large, deep neural networks to convergence can be prohibitivelyexpensive. As a result, often only a small selection of popular, dense modelsare reused across different contexts and tasks. Increasingly, sparselyactivated models, which seek to decouple model size from computation costs, arebecoming an attractive alternative to dense models. Although more efficient interms of quality and computation cost, sparse models remain data-hungry andcostly to train from scratch in the large scale regime. In this work, wepropose sparse upcycling -- a simple way to reuse sunk training costs byinitializing a sparsely activated Mixture-of-Experts model from a densecheckpoint. We show that sparsely upcycled T5 Base, Large, and XL languagemodels and Vision Transformer Base and Large models, respectively,significantly outperform their dense counterparts on SuperGLUE and ImageNet,using only ~50% of the initial dense pretraining sunk cost. The upcycled modelsalso outperform sparse models trained from scratch on 100% of the initial densepretraining computation budget.