Abstract
This paper explores a new research problem of unsupervised transfer learningacross multiple spatiotemporal prediction tasks. Unlike most existing transferlearning methods that focus on fixing the discrepancy between supervised tasks,we study how to transfer knowledge from a zoo of unsupervisedly learned modelstowards another predictive network. Our motivation is that models fromdifferent sources are expected to understand the complex spatiotemporaldynamics from different perspectives, thereby effectively supplementing the newtask, even if the task has sufficient training samples. Technically, we proposea differentiable framework named transferable memory. It adaptively distillsknowledge from a bank of memory states of multiple pretrained RNNs, and appliesit to the target network via a novel recurrent structure called theTransferable Memory Unit (TMU). Compared with finetuning, our approach yieldssignificant improvements on three benchmarks for spatiotemporal prediction, andbenefits the target task even from less relevant pretext ones.