Abstract
The prominence of deep learning, large amount of annotated data andincreasingly powerful hardware made it possible to reach remarkable performancefor supervised classification tasks, in many cases saturating the trainingsets. However, adapting the learned classification to new domains remains ahard problem due to at least three reasons: (1) the domains and the tasks mightbe drastically different; (2) there might be very limited amount of annotateddata on the new domain and (3) full training of a new model for each new taskis prohibitive in terms of memory, due to the shear number of parameter of deepnetworks. Instead, new tasks should be learned incrementally, building on priorknowledge from already learned tasks, and without catastrophic forgetting, i.e.without hurting performance on prior tasks. To our knowledge this paperpresents the first method for multi-domain/task learning without catastrophicforgetting using a fully tensorized architecture. Our main contribution is amethod for multi-domain learning which models groups of identically structuredblocks within a CNN as a high-order tensor. We show that this joint modellingnaturally leverages correlations across different layers and results in morecompact representations for each new task/domain over previous methods whichhave focused on adapting each layer separately. We apply the proposed method to10 datasets of the Visual Decathlon Challenge and show that our method offerson average about 7.5x reduction in number of parameters and superiorperformance in terms of both classification accuracy and Decathlon score. Inparticular, our method outperforms all prior work on the Visual DecathlonChallenge.