Abstract
Despite the recent success of multi-task learning and transfer learning fornatural language processing (NLP), few works have systematically studied theeffect of scaling up the number of tasks during pre-training. Towards thisgoal, this paper introduces ExMix (Extreme Mixture): a massive collection of107 supervised NLP tasks across diverse domains and task-families. Using ExMix,we study the effect of multi-task pre-training at the largest scale to date,and analyze co-training transfer amongst common families of tasks. Through thisanalysis, we show that manually curating an ideal set of tasks for multi-taskpre-training is not straightforward, and that multi-task scaling can vastlyimprove models on its own. Finally, we propose ExT5: a model pre-trained usinga multi-task objective of self-supervised span denoising and supervised ExMix.Via extensive experiments, we show that ExT5 outperforms strong T5 baselines onSuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside ofExMix. ExT5 also significantly improves sample efficiency while pre-training.