Abstract
Recurrent Neural Networks (RNNs) are powerful sequence modeling tools.However, when dealing with high dimensional inputs, the training of RNNsbecomes computational expensive due to the large number of model parameters.This hinders RNNs from solving many important computer vision tasks, such asAction Recognition in Videos and Image Captioning. To overcome this problem, wepropose a compact and flexible structure, namely Block-Term tensordecomposition, which greatly reduces the parameters of RNNs and improves theirtraining efficiency. Compared with alternative low-rank approximations, such astensor-train RNN (TT-RNN), our method, Block-Term RNN (BT-RNN), is not onlymore concise (when using the same rank), but also able to attain a betterapproximation to the original RNNs with much fewer parameters. On threechallenging tasks, including Action Recognition in Videos, Image Captioning andImage Generation, BT-RNN outperforms TT-RNN and the standard RNN in terms ofboth prediction accuracy and convergence rate. Specifically, BT-LSTM utilizes17,388 times fewer parameters than the standard LSTM to achieve an accuracyimprovement over 15.6\% in the Action Recognition task on the UCF11 dataset.