Abstract
Pre-training models on Imagenet or other massive datasets of real images hasled to major advances in computer vision, albeit accompanied with shortcomingsrelated to curation cost, privacy, usage rights, and ethical issues. In thispaper, for the first time, we study the transferability of pre-trained modelsbased on synthetic data generated by graphics simulators to downstream tasksfrom very different domains. In using such synthetic data for pre-training, wefind that downstream performance on different tasks are favored by differentconfigurations of simulation parameters (e.g. lighting, object pose,backgrounds, etc.), and that there is no one-size-fits-all solution. It is thusbetter to tailor synthetic pre-training data to a specific downstream task, forbest performance. We introduce Task2Sim, a unified model mapping downstreamtask representations to optimal simulation parameters to generate syntheticpre-training data for them. Task2Sim learns this mapping by training to findthe set of best parameters on a set of "seen" tasks. Once trained, it can thenbe used to predict best simulation parameters for novel "unseen" tasks in oneshot, without requiring additional training. Given a budget in number of imagesper class, our extensive experiments with 20 diverse downstream tasks showTask2Sim's task-adaptive pre-training data results in significantly betterdownstream performance than non-adaptively choosing simulation parameters onboth seen and unseen tasks. It is even competitive with pre-training on realimages from Imagenet.