Abstract
Recent findings indicate that overparametrization, while crucial forsuccessfully training deep neural networks, also introduces large amounts ofredundancy. Tensor methods have the potential to efficiently parametrizeovercomplete representations by leveraging this redundancy. In this paper, wepropose to fully parametrize Convolutional Neural Networks (CNNs) with a singlehighorder, lowrank tensor. Previous works on network tensorization havefocused on parametrizing individual layers (convolutional or fully connected)only, and perform the tensorization layerbylayer separately. In contrast, wepropose to jointly capture the full structure of a neural network byparametrizing it with a single highorder tensor, the modes of which representeach of the architectural design parameters of the network (e.g. number ofconvolutional blocks, depth, number of stacks, input features, etc). Thisparametrization allows to regularize the whole network and drastically reducethe number of parameters. Our model is endtoend trainable and the lowrankstructure imposed on the weight tensor acts as an implicit regularization. Westudy the case of networks with rich structure, namely Fully ConvolutionalNetworks (FCNs), which we propose to parametrize with a single 8thordertensor. We show that our approach can achieve superior performance with smallcompression rates, and attain high compression rates with negligible drop inaccuracy for the challenging task of human pose estimation.
Quick Read (beta)
TNet: Parametrizing Fully Convolutional Nets with a Single HighOrder Tensor
Abstract
Recent findings indicate that overparametrization, while crucial for successfully training deep neural networks, also introduces large amounts of redundancy. Tensor methods have the potential to efficiently parametrize overcomplete representations by leveraging this redundancy. In this paper, we propose to fully parametrize Convolutional Neural Networks (CNNs) with a single highorder, lowrank tensor. Previous works on network tensorization have focused on parametrizing individual layers (convolutional or fully connected) only, and perform the tensorization layerbylayer separately. In contrast, we propose to jointly capture the full structure of a neural network by parametrizing it with a single highorder tensor, the modes of which represent each of the architectural design parameters of the network (e.g. number of convolutional blocks, depth, number of stacks, input features, etc). This parametrization allows to regularize the whole network and drastically reduce the number of parameters. Our model is endtoend trainable and the lowrank structure imposed on the weight tensor acts as an implicit regularization. We study the case of networks with rich structure, namely Fully Convolutional Networks (FCNs), which we propose to parametrize with a single $8$${}^{\text{th}}$–order tensor. We show that our approach can achieve superior performance with small compression rates, and attain high compression rates with negligible drop in accuracy for the challenging task of human pose estimation.
1 Introduction
For a wide range of challenging tasks, including recognition [22, 34, 13], detection [32], semantic segmentation [25, 12] and human pose estimation [26], the stateoftheart is currently attained with Convolutional Neural Networks (CNNs). There is evidence that a key feature behind the success of these methods is overparametrization, which helps find good local minima [11, 35]. However, at the same time, overparametrization leads to a great amount of redundancy, and from a statistical perspective, it might hinder generalization because it excessively increases the number of parameters. Furthermore, models with an excessive number of parameters have increased storage and computation requirements, rendering them problematic for deployment on devices with limited computational resources. This paper focuses on a novel way of leveraging the redundancy in the parameters of CNNs by jointly parametrizing the whole network using tensor methods.
There is a significant amount of recent work on using tensors to reduce the redundancy and improve the efficiency of CNNs, mostly focusing on reparametrizing individual layers. For example, [36, 17] treat a convolutional layer as a 4D tensor and then compute a decomposition of the 4D tensor into a sum of a small number of lowrank tensors. Similarly, [27] proposes tensorizing the fullyconnected layers. The bulk of these methods focus on tensorizing individual layers only, and perform the tensorization layerbylayer disjointly, usually by applying a tensor decomposition to the pretrained weights and then finetuning to compensate for the performance loss. For example, [36] tensorizes the second convolutional layer of AlexNet [22].
Our paper primarily departs from prior work by using a single highorder tensor to parametrize the whole CNN as opposed to using different tensors to parametrize the individual layers. In particular, we propose to parametrize the network with a single highorder tensor, each dimension of which represents a different architectural design parameter of the network. For the case of Fully Convolutional Networks (FCNs) with an encoderdecoder structure considered herein (see also Fig. 1), each dimension of the $8$dimensional tensor represents a different architectural design parameter of the network such as the number of (stacked) FCNs used, the depth of each network, the number of input and output features for each convolutional block and the spatial dimensions of each of the convolutional kernels. By modelling the whole FCN with a single tensor, our approach allows for learning correlations between the different tensor dimensions and hence to fully capture the structure of the network. Moreover, this parametrization implicitly regularizes the whole network and drastically reduces the number of parameters by imposing a lowrank structure on that tensor. Owing to these properties, our framework is much more general and flexible compared to prior work offering increased accuracy and high compression rates. In summary, the contributions of this work are:

•
We propose using a single highorder tensor for whole network tensorization and applying it for capturing the rich structure of Fully Convolutional Networks. Our endtoend trainable approach allows for a wide spectrum of network decompositions and compression rates which can be chosen and optimized for a particular application.

•
We show that for a large range of compression rates (both high and low), our method preserves high accuracy. Compared to prior work based on tensorizing individual convolutional layers, our method consistently achieves higher accuracy, especially for the case of high compression rates. In addition, we show that, for lower compression rates, our method outperforms the original uncompressed network.

•
We illustrate the favorable properties of our method by performing a large number of experiments and ablation studies for the challenging task of human pose estimation. The experiments shed light on several interesting aspects of our method including the effect of varying the rank for each mode of the tensor, as well as the decomposition method used. We further validate our conclusions on a different dense prediction task, namely semantic facial part segmentation.
2 Related Work
In this section, we review related work, both for tensor methods and human pose estimation.
Tensor methods
offer a natural extension of traditional algebraic methods to higher orders. For instance, Tucker decomposition can be seen as a generalization of PCA to higher dimensions [18]. Tensor decompositions have widereaching applications, including learning a wide range of probabilistic latentvariable models [1]. Tensor methods have been recently applied to deep learning, for instance, to provide a theoretical analysis of deep neural nets [9]. New layers were also proposed, leveraging tensor methods. [19] proposes tensor contraction layers to reduce the dimensionality of activation tensors while preserving their multilinear structure. Tensor regression layers [20] express outputs through a lowrank multilinear mapping from a highorder activation tensor to an output tensor of arbitrary order.
A lot of existing work has been dedicated to leveraging tensor decompositions in order to reparametrizing existing layers, either to speed up computation or to reduce the number of parameters. Separable convolutions, for instance, can be obtained from existing ones by applying CP decomposition to their kernel. The authors in [23] propose such parametrization of individual convolutional layers using CP decomposition with the goal of speeding them up. Specifically, each of the 4D tensors parametrizing the convolutional layers of a pretrained network are decomposed into a sum of rank–$1$ tensors using CP decomposition. The resulting factors are used to replace each existing convolution with a series of $4$ convolutional layers with smaller kernels. The network is then finetuned to restore performance. [17] proposes a similar approach but uses Tucker decomposition instead of CP to decompose the convolutional layers of a pretrained network, before finetuning to restore the performance. Specifically, Tucker decomposition is applied to each convolutional kernel of a pretrained network, on two of the modes (input and output channel modes). The resulting network is finetuned to compensate for the drop in performance induced by the compression.
In [3], the layers of deep convolutional neural networks are also reparametrized using CP decomposition, optimized using the tensor power method. The network is then iteratively finetuned to restore performance. Similarly, [36] proposes to use tensor decomposition to remove redundancy in convolutional layers and express these as the composition of two convolutional layers with less parameters. Each 2D filter is approximated by a sum of rank–$1$ matrices. Thanks to this restricted setting, a closedform solution can be readily obtained with SVD. This is done for each convolutional layer with a kernel of size larger than 1. While all these focus of convolutional layers, other types of layers can also be parametrized. For instance, [27] uses the TensorTrain (TT) format [28] to impose a lowrank tensor structure on the weights of the fullyconnected layers. Tensorization of generative adversarial networks [6] and sequence models [45] have been also proposed.
The work of [7] proposes a new residual block, the socalled collective residual unit (CRU), which is obtained by applying a generalized block term decomposition to the last two modes of a 4${}^{\text{th}}$–order tensor obtained by stacking the convolutional kernels of several residual units. Similarly to existing works, each of the CRUs is parametrized individually. [44] leverages tensor decomposition for multitask learning to allow for weight sharing between the fullyconnected and convolutional layers of two or more deep neural networks.
Overall, to the best of our knowledge, our work is the first to propose an endtoend trainable architecture, fully parametrized by a single high order lowrank tensor.
Other methods for network decomposition.
There are also other methods, besides tensorbased ones, for reducing the redundancy and number of parameters in neural networks. A popular approach is quantization which is concerned with quantizing the weights and/or the features of a network [41, 43, 46, 37, 10, 31]. Quantization methods should be considered orthogonal to tensor methods as one could apply them to the output of tensor decompositions, too. Similarly, complementary to our work should be considered methods for improving the efficiency of neural networks using weight pruning [24, 14].
More related to our work are handcrafted decomposition methods such as MobileNet [15] and Xception [8] which decompose $3\times 3$ convolutions using efficient depthwise and pointwise convolutions. We compare our methods with MobileNet, the method of choice for improving the efficiency of CNNs, and show that our approach outperforms it by large margin.
Human pose estimation.
CNN–based methods have recently produced results of remarkable accuracy for the task of human pose estimation, outperforming traditional methods by large margin [40, 39, 30, 4, 26, 42]. Arguably, one of the most widely used architectures for this task is the stacked HourGlass (HG) network proposed by [26]. An HG is an encoderdecoder network with skip connections between the encoder and the decoder, suitable for making predictions at a pixel level in a fully convolutional manner. [26] uses a stack of 8 of these networks to achieve stateoftheart performance on the MPII dataset [2]. The architecture is shown in Fig. 1. In this work, we choose tensorizing the HG network primarily because of its rich structure which makes it suitable to model it with a highorder tensor. We note that the aim of this work is not to produce stateoftheart results for the task of human pose estimation but to show the benefits of modelling a stateoftheart architecture with a single highorder tensor.
3 Mathematical background
In this section we first introduce some mathematical background regarding the notation and tensor methods used in this paper.
Notation.
We denote vectors (1${}^{\text{st}}$–order tensors) as $\mathbf{v}$, matrices (2${}^{\text{nd}}$–order tensors) as $\mathbf{M}$, and tensors of order 3 or greater as $\mathcal{X}$ . We denote element $({i}_{0},{i}_{1},\mathrm{\cdots},{i}_{N})$ of a tensor as ${\mathcal{X}}_{{i}_{0},{i}_{1},\mathrm{\cdots},{i}_{N}}\text{or}\mathcal{X}({i}_{0},{i}_{1},\mathrm{\cdots},{i}_{N})$. A colon is used to denote all elements of a mode, e.g. the mode–1 fibers of $\mathcal{X}$ are denoted as $\mathcal{X}(:,{i}_{2},{i}_{3},\mathrm{\cdots},{i}_{N})$. Finally, for any $i,j\in \mathbb{N},[i..j]$ denotes the set of integers $\{i,i+1,\mathrm{\cdots},j1,j\}$.
Mode–$n$ unfolding
of a tensor $\mathcal{X}\in {\mathbb{R}}^{{I}_{0}\times {I}_{1}\times \mathrm{\cdots}\times {I}_{N}}$, is a matrix ${\mathbf{X}}_{[n]}\in {\mathbb{R}}^{{I}_{n},{I}_{M}}$, with $M={\prod}_{\begin{array}{c}k=0,\\ k\ne n\end{array}}^{N}{I}_{k}$, defined by the mapping from element $({i}_{0},{i}_{1},\mathrm{\cdots},{i}_{N})$ to $({i}_{n},j)$, with $j={\sum}_{\begin{array}{c}k=0,\\ k\ne n\end{array}}^{N}{i}_{k}\times {\prod}_{\begin{array}{c}m=k+1,\\ m\ne n\end{array}}^{N}{I}_{m}$.
Moden product.
For a tensor $\mathcal{X}\in {\mathbb{R}}^{{I}_{0}\times {I}_{1}\times \mathrm{\cdots}\times {I}_{N}}$ and a matrix $\mathbf{M}\in {\mathbb{R}}^{R\times {I}_{n}}$, the nmode product of a tensor is a tensor of size $\left({I}_{0}\times \mathrm{\cdots}\times {I}_{n1}\times R\times {I}_{n+1}\times \mathrm{\cdots}\times {I}_{N}\right)$ and can be expressed using the unfolding of $\mathcal{X}$ and the classical dot product as $\mathcal{X}{\times}_{n}\mathbf{M}=\mathbf{M}{\mathcal{X}}_{[n]}\in {\mathbb{R}}^{{I}_{0}\times \mathrm{\cdots}\times {I}_{n1}\times R\times {I}_{n+1}\times \mathrm{\cdots}\times {I}_{N}}$
Tensor diagrams.
While ${2}^{\text{nd}}$–order tensors can easily be depicted as rectangles and ${3}^{\text{rd}}$–order tensors as cubes, it is impractical to represent highorder tensors in such way. We instead use tensor diagrams, which are undirected graphs where the vertices represent tensors. The degree of each vertex (i.e. the number of edges originating from this circle) specifies the order of the corresponding tensor. Tensor contraction over two modes is then represented by simply linking together the two edges corresponding to these two modes. Fig. 2 depicts the Tucker decomposition (i.e. contraction of a core tensor with factor matrices along each mode) of an 8${}^{\text{th}}$–order tensor with tensor diagrams.
4 TNet: Fullytensorized FCN architecture
In this section, we introduce our fullytensorized method by first introducing the architecture before detailing the structure of the parametrization weights.
4.1 FCN tensorization
In this section, we describe how to tensorize the stacked HourGlass (HG) architecture of [26]. The HG has a number of design parameters namely the number of (stacked) HGs, the depth of each HG, the three signal pathways of each HG (skip, downsample and upsample), the number of convolutional layers in each residual block (i.e. the depth of each block), the number of input and output features of each block and finally, the spatial dimensions of each of the convolutional kernels.
To facilitate the tensorization of the whole network, we used a modified HG architecture in which we replaced all the residual modules with the basic block introduced by [13], maintaining the same number of input and output channels throughout the network. We made the encoder and the decoder symmetric, with 4 residual modules each. Figure 1 illustrates the modified HG architecture. We note that from an accuracy perspective, this modification performs (almost) the same as the original HG proposed in [26].
From the network described above, we derive the highorder tensor for the proposed TensorizedNetwork (TNet) as follows: all weights of the network are parametrized by a single ${8}^{\text{th}}$–order tensor $\mathcal{W}\in {\mathbb{R}}^{{I}_{0}\times {I}_{1}\times \mathrm{\cdots}\times {I}_{7}}$, the modes of which correspond to the number of HGs (${I}_{0}=$ #hg), the depth of each HG (${I}_{1}=$ hg${}_{\text{depth}}$), the three signal pathways (${I}_{2}=$ hg${}_{\text{subnet}}$), the number of convolutional layers per block (${I}_{3}=$ b${}_{\text{depth}}$), the number of input features (${I}_{4}=$ f${}_{\text{in}}$), the number of output features (${I}_{5}=$ f${}_{\text{out}}$), and finally the height (${I}_{6}=$ h) and width (${I}_{7}=$ w) of each of the convolutional kernels.
4.2 TNet variants
Based on the previous parametrization of the network, we can add various lowrank constraints on the weight tensor, leading to variants of our method.
Tucker TNet.
The Tucker form of our model expresses the constructed 8${}^{\text{th}}$–order tensor $\mathcal{W}$ as a rank–$({R}_{0},\mathrm{\cdots},{R}_{7})$ Tucker tensor, composed of a low rank core $\mathcal{G}\in {\mathbb{R}}^{{R}_{0}\times {R}_{1}\times \mathrm{\cdots}\times {R}_{7}}$ along with projection factors $({\mathbf{U}}^{(0)},\mathrm{\cdots},{\mathbf{U}}^{(7)})$, with ${\mathbf{U}}^{(k)}\in {\mathbb{R}}^{{R}_{k},{I}_{k}},k\in [0..7]$. This allows us to write the network’s weight tensor in a decomposed form as:
$\mathcal{W}$  $=\mathcal{G}{\times}_{0}{\mathbf{U}}^{(0)}{\times}_{1}{\mathbf{U}}^{(2)}\times \mathrm{\cdots}{\times}_{7}{\mathbf{U}}^{(7)}$  (1)  
$=\u27e6\mathcal{G};{\mathbf{U}}^{(0)},\mathrm{\cdots},{\mathbf{U}}^{(7)}\u27e7$ 
See also Fig. 2 for a tensor diagram of the Tucker form of the weight tensor. Note that the CP decomposition is the special case of the Tucker decomposition, where the core is superdiagonal.
MPS TNet.
The MatrixProductState (MPS) form (also known as tensortrain [28]) of our model expresses the constructed 8${}^{\text{th}}$–order weight tensor $\mathcal{W}$ as a series of thirdorder tensors (the cores) and allows for especially large spacesavings. In our case, given $\mathcal{W}\in {\mathbb{R}}^{{I}_{0}\times {I}_{1}\times \mathrm{\cdots}\times {I}_{7}}$, we can decompose it into a rank $({R}_{0},{R}_{1},\mathrm{\cdots}{R}_{8})$–MPS as a series of thirdorder cores ${\mathcal{G}}_{0}\in {\mathbb{R}}^{{R}_{0},{I}_{0},{R}_{1}},{\mathcal{G}}_{1}\in {\mathbb{R}}^{{R}_{1},{I}_{1},{R}_{2}},\mathrm{\cdots},{\mathcal{G}}_{7}\in {\mathbb{R}}^{{R}_{7},{I}_{7},{R}_{8}}$. The boundary conditions dictate ${R}_{0}={R}_{8}=1$. In terms of individual elements, we can then write, for any ${i}_{0}\in [0..{I}_{0}],{i}_{1}\in [0..{I}_{1}],\mathrm{\cdots},{i}_{7}\in [0..{I}_{7}]$:
$$\mathcal{W}({i}_{0},{i}_{1},\mathrm{\cdots},{i}_{7})=\underset{1\times {R}_{1}}{\underset{\u23df}{{\mathcal{G}}_{0}[{i}_{0}]}}\times \underset{{R}_{1}\times {R}_{2}}{\underset{\u23df}{{\mathcal{G}}_{1}[{i}_{1}]}}\times \mathrm{\cdots}\times \underset{{R}_{7}\times 1}{\underset{\u23df}{{\mathcal{G}}_{7}[{i}_{7}]}}$$ 
4.3 Parameter analysis
This section compares the number of parameters of our model which parameterizes the whole weight tensor with a single highorder tensor with methods based on layerwise decomposition (e.g. [17, 23]). Considering a Tucker rank–${R}_{0},{R}_{1},\mathrm{\cdots},{R}_{7}$ of the weight tensor parametrizing the whole network, the resulting number of parameters is:
$${N}_{TNet}=\prod _{k=0}^{7}{R}_{k}+\sum _{k=0}^{7}{R}_{k}\times {I}_{k}.$$  (2) 
Compressing each of the ${N}_{\text{conv}}$ convolutional layer separately [17], with a rank ${R}_{4}$ and ${R}_{5}$ for the number of input and output features, respectively, and writing ${N}_{\text{conv}}={\prod}_{k=0}^{4}{I}_{k}$, we obtain the total number of parameters:
$${N}_{\text{conv}}\times \left({R}_{4}\times {R}_{5}\times {I}_{6}\times {I}_{7}+{R}_{4}\times {I}_{4}+{R}_{5}{I}_{5}\right).$$  (3) 
In comparison, our model, with the same ranks ${R}_{4}$ and ${R}_{5}$ imposed on the number of features, would only have ${N}_{\text{conv}}\times \left({R}_{4}\times {R}_{5}\times 3\times 3\right)+{R}_{4}\times {I}_{4}+{R}_{5}{I}_{5}$ parameters. In other words, our model has $\left({\prod}_{k=0}^{4}{I}_{k}1\right)\left({R}_{4}\times {I}_{4}+{R}_{5}{I}_{5}\right)$ parameters less than a corresponding layerwise decomposition.
Speeding up the convolutions.
When parametrized using a CP or Tucker decomposition, a convolutional layer can be efficiently replaced by a series of convolutions with smaller kernels [23, 17], thus allowing for large computational speedups. This efficient reparametrization also applies to our model. To see this, given the weight tensor $\mathcal{W}\in {\mathbb{R}}^{{I}_{0}\times {I}_{1}\times \mathrm{\cdots}\times {I}_{7}}$ of our Tucker TNet, we have
$$\mathcal{W}=\mathcal{G}{\times}_{0}{\mathbf{U}}^{(0)}{\times}_{1}{\mathbf{U}}^{(1)}\times \mathrm{\cdots}{\times}_{7}{\mathbf{U}}^{(7)}.$$ 
For any ${i}_{0},{i}_{1},{i}_{2},{i}_{3}\in ({I}_{0},{I}_{1},{I}_{2},{I}_{3})$, let us denote $\stackrel{~}{\mathcal{K}}=\mathcal{W}({i}_{0},{i}_{1},{i}_{2},{i}_{3},\mathbf{:},\mathbf{:},\mathbf{:},\mathbf{:})$, corresponding to one of the convolutional kernels of the TNet. By rearranging the terms, and considering the partially contracted core, we can write:
$$\stackrel{~}{\mathcal{K}}(s,t,j,k)=\sum _{{r}_{4}=0}^{{R}_{4}}\sum _{{r}_{5}=0}^{{R}_{5}}\mathcal{C}({r}_{4},{r}_{5},j,k){\mathbf{U}}^{(4)}(s,{r}_{4}){\mathbf{U}}^{(5)}(t,{r}_{5})$$ 
with $\mathcal{C}={\mathcal{C}}_{{i}_{0},{i}_{1},{i}_{2},{i}_{3},\mathbf{:},\mathbf{:},\mathbf{:},\mathbf{:}}\in {\mathbb{R}}^{({I}_{4},{I}_{5},{I}_{6},{I}_{7})}$ and
$$\mathcal{C}=\left(\mathcal{G}{\times}_{0}{\mathbf{U}}^{(0)}\times \mathrm{\cdots}{\times}_{3}{\mathbf{U}}^{(3)}{\times}_{6}{\mathbf{U}}^{(6)}{\times}_{7}{\mathbf{U}}^{(7)}\right).$$ 
This gives us an effective way of approximating each convolution by three smaller convolutions [17]. While getting the full speedup would require the writing of specialized CUDA kernels, some timings results with a naive implementation using PyTorch are shown in Table 1, for a single convolutional layer with a kernel tensor of size $128\times 128\times 3\times 3$ compressed using Tucker decomposition.
Baseline  Tucker $\mathrm{1.37}\mathrm{\times}$  Tucker $\mathrm{2.77}\mathrm{\times}$  Tucker $\mathrm{4.17}\mathrm{\times}$ 
3.79 ms.  4.36 ms.  2.72 ms.  2.45 ms. 
5 Experimental Setup
Tuckerrank  Accuracy  Compression  
#hg  hg${}_{\text{\mathbf{d}\mathbf{e}\mathbf{p}\mathbf{t}\mathbf{h}}}$  hg${}_{\text{\mathbf{s}\mathbf{u}\mathbf{b}\mathbf{n}\mathbf{e}\mathbf{t}}}$  b${}_{\text{\mathbf{d}\mathbf{e}\mathbf{p}\mathbf{t}\mathbf{h}}}$  f${}_{\text{\mathbf{i}\mathbf{n}}}$  f${}_{\text{\mathbf{o}\mathbf{u}\mathbf{t}}}$  h  w  (PCKh)  ratio 
Original  86.99%  1.0x  
3  4  3  2  128  128  3  3  87.42%  1.28x 
2  4  3  2  128  128  3  3  86.95%  1.82x 
1  4  3  2  128  128  3  3  86.05%  3.03x 
4  3  3  2  128  128  3  3  87.71%  1.28x 
4  2  3  2  128  128  3  3  87.59%  1.82x 
4  1  3  2  128  128  3  3  86.89%  3.03x 
4  4  2  2  128  128  3  3  87.53%  1.43x 
4  4  1  2  128  128  3  3  86.19%  2.50x 
4  4  3  1  128  128  3  3  82.59%  1.82x 
4  4  3  2  96  96  3  3  87.43%  1.64x 
4  4  3  2  64  64  3  3  86.13%  3.03x 
4  4  3  2  32  32  3  3  83.10%  6.25x 
4  4  3  2  128  128  2  2  87.30%  1.98x 
The bulk of our experiments were conducted for the task of human pose estimation. We also validated some of our conclusions by conducting experiments for a different dense prediction task, namely facial part segmentation.
Human pose estimation.
Semantic facial part segmentation.
Implementation details
We used a stacked HG architecture with the following architectural parameters: #hg = 4, hg${}_{\text{depth}}$ = 4, hg${}_{\text{subnet}}$=3, b${}_{\text{depth}}$ = 2, f${}_{\text{in}}$=128, f${}_{\text{out}}$=128, and h = w = 3. This resulted in a ${8}^{\text{th}}$–order tensor of size $4\times 4\times 3\times 2\times 128\times 128\times 3\times 3$.
For the uncompressed baseline network, we reduced the number of its parameters by simply decreasing the number of channels in each residual block, varying it from 128 to 64. By doing so, (as opposed to reducing the number of stacks), we maintain all the architectural advantages offered by the stacked HG architecture and ensure a fair comparison with the proposed tensorized network.
Training.
All models were trained for 110 epochs using RMSprop [38]. The learning rate was varied from $2.5e4$ to $1e6$ using a MultiStep fixed scheduler. During training, we randomly augmented the data using: rotation (${25}^{\circ}$ to ${25}^{\circ}$ for human pose and ${40}^{\circ}$ to ${40}^{\circ}$ for face part segmentation), scale distortion ($0.75$ to $1.25$), horizontal flipping and color jittering.
All experiments were run on a single NVIDIA TITAN V GPU. All networks were implemented using PyTorch [29]. TensorLy [21] was used for all tensor operations.
Method  Parameters  Compression ratio  Accuracy 

Uncompressed Baseline  full, f${}_{\text{in}}$=f${}_{\text{out}}$=128  1x  87% 
Trimmed Baseline  f${}_{\text{in}}$=f${}_{\text{out}}$=112  1.3x  86.9% 
Trimmed Baseline  f${}_{\text{in}}$=f${}_{\text{out}}$=92  2x  85.9% 
Trimmed Baseline  f${}_{\text{in}}$=f${}_{\text{out}}$=64  4x  84.5% 
Trimmed Baseline  hg_depth=3  1.3x  86.79% 
Trimmed Baseline  hg_depth=2  1.8x  86.82% 
Trimmed Baseline  hg_depth=1  3.0x  85.30% 
MobileNet[16]  f${}_{\text{in}}$=f${}_{\text{out}}$=194  3.6x  84.3% 
MobileNet[16]  f${}_{\text{in}}$=f${}_{\text{out}}$=160  5.4x  82.7% 
[17]  rank–(128, 128, 2, 2)  1.4x  84.9% 
[17]  rank–(96, 96, 3, 3)  1.3x  86.8% 
[17]  rank–(64, 64, 3, 3)  2.3x  86.4% 
[17]  rank–(32, 32, 3, 3)  4.7x  85.3% 
[17]  rank–(16, 16, 3, 3)  6.9x  83.7% 
Tucker TNet [Ours]  rank–$(4,3,3,2,110,110,3,3)$  1.7x  87.5% 
Tucker TNet [Ours]  rank–$(4,4,2,2,110,110,3,3)$  1.8x  87.4% 
Tucker TNet [Ours]  rank–$(3,3,3,2,110,110,2,2)$  3.7x  87.1% 
Tucker TNet [Ours]  rank–$(3,2,3,2,96,96,3,3)$  3.4x  86.7% 
Tucker TNet [Ours]  rank–$(3,3,2,2,80,80,3,3)$  4.2x  86.3% 
Tucker TNet [Ours]  rank–$(2,2,2,2,96,96,3,3)$  5.2x  86.0% 
MPS TNet [Ours]  rank–$(1,4,4,12,24,110,9,3,1)$  7.4x  85.5% 
Performance measures.
For the human pose estimation experiments, we report accuracy in terms of PCKh [2]. For facial part segmentation, we report segmentation accuracy using the mean accuracy and mIOU metrics [25]. Finally, we measure the parameter savings using the $\text{compression ratio}=\frac{\text{uncompressed}}{\text{compressed}}$, defined as the total number of parameters of the uncompressed network divided by the number of parameters of the compressed network.
6 Results
This section offers an indepth analysis of the performance and accuracy of the proposed TNet. Our main results are that the proposed approach: i) outperforms the layerwise decomposition of [17] and [23], which are the most closely related works to our method; ii) outperforms the uncompressed, original network for low compression rates; iii) achieves consistent compression ratios across arbitrary dimensions and iv) outperforms MobileNet [16] by large margin. Finally, we further validate some of these results for the task of semantic facial part segmentation.
All results reported were obtained by finetuning our networks in an endtoend manner from a pretrained uncompressed original network. We were able to reach the same level of accuracy when training from scratch, though this required training for more iterations. In contrast, we found that when trained from scratch, the layerwise method of [17] reaches subpar performance, as also reported in their paper.
Method  Parameters  Compression ratio  mIOU  mAcc 

Uncompressed baseline  full, f${}_{\text{in}}$=f${}_{\text{out}}$=128  1x  76.02%  97.31% 
TNet [Ours]  Tucker–$(3,2,3,2,96,96,3,3)$  3.38x  76.01%  97.29% 
TNet [Ours]  Tucker–$(2,2,2,2,64,64,3,3)$  6.94x  75.57%  97.01% 
6.1 Redundancy of the weight tensor
In order to better understand the compressibility of each mode of the weight tensor, we first investigate the redundancy of each of the modes of the tensor by compressing only one of the modes at a time. Table 2 shows the accuracy (PCKh) as well as the compression ratio obtained by compressing one of the modes, corresponding respectively to the number of HGs (#hg), the depth of each HG (hg${}_{\text{depth}}$), the three pathways of each HG (hg${}_{\text{subnet}}$), the number of convolutional layers per blocks (b${}_{\text{depth}}$) and, finally, the number of input features (f${}_{\text{in}}$), output features (f${}_{\text{out}}$), height (h) and the width (w) of each of the convolutional kernels. The results are shown along with the performance of the original uncompressed network. We observe that by taking advantage of the redundancy at networklevel (as opposed to [23, 17] which compress individual layers), the proposed approach is able to effectively compress across arbitrary dimensions for large compression ratios while maintaining similar, or even in some cases higher, accuracy than that of the original uncompressed network.
6.2 Performance of the TNet
Based on the insights gained from the previous experiment, we selected promising configurations and compressed over multiple dimensions simultaneously. We then compared these configurations with baseline and stateoftheart methods. The results can be seen in Table 3.
Compression vs. trimming. The obvious comparison is between TNet and the original baseline network, “compressed” by trimming it, reducing the number of parameters to match the compression ratio achieved by TNet.
Comparison with efficient architectures. A natural question is whether TNet performs favourably when compared to architectures designed for efficiency. To answer this, we performed a comparison with MobileNet [16], for which we adjusted the number of channels of the convolutional layers in order to vary the number of parameters and obtain comparable compression ratios.
Comparison with the stateoftheart. We also compared with the layerwise decomposition method of [17].
We firstly observe that by just reducing the number of channels in the original network, a significant drop in performance can be noticed. Secondly, our method consistently outperforms [17] across the whole spectrum of compression ratios. This can be seen by comparing the accuracy provided for any compression ratio for [17] with the accuracy of the closest but higher compression ratio for our method (for example, compare $2.33$x for [17] with $3.67$x for our method). Our method always achieves higher accuracy even though the compression ratio is also higher. In addition, unlike to [17] which does not seem to work well when the size of the convolutional kernel is compressed from $3\times 3$ to $2\times 2$, our method is able to compress that dimension too while maintaining similar level of accuracy. Finally, our method outperforms MobileNet [16] by a large margin.
In the same table, we also report the performance of a variant of our method, using an MPS decomposition on the weights rather than a Tucker one. This result shows that our method works effectively with other decomposition methods as well. Nevertheless, we focused mainly on Tucker as it is the most flexible compression method, allowing us to control the rank of each mode of the weight tensor.
Results on face segmentation. Finally, we selected two of our best performing models and retrained them for the task of semantic facial part segmentation. Our method offers significant compression ratios (up to 7x) with virtually no loss in accuracy (see Table 4). These results further confirm that our method is taskindependent.
7 Conclusions
We proposed an endtoend trainable method to jointly capture the full structure of a fullyconvolutional neural network, by parametrizing it with a single, highorder lowrank tensor. The modes of this tensor represent each of the architectural design parameters of the network (e.g. number of convolutional blocks, depth, number of stacks, input features, etc). This parametrization allows for a joint regularization of the whole network. The number of parameters can be drastically reduced by imposing a lowrank structure on the parameter tensor. We show that our approach can achieve superior performance with low compression rates, and attain high compression rates with negligible drop in accuracy, on both the challenging task of human pose estimation and semantic face segmentation.
References
 [1] Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M. Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. JMLR, 15(1):2773–2832, jan 2014.
 [2] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014.
 [3] Marcella Astrid and SeungIk Lee. Cpdecomposition with tensor power method for convolutional neural networks compression. CoRR, abs/1701.07148, 2017.
 [4] Adrian Bulat and Georgios Tzimiropoulos. Human pose estimation via convolutional part heatmap regression. In ECCV, 2016.
 [5] Adrian Bulat and Georgios Tzimiropoulos. Binarized convolutional landmark localizers for human pose estimation and face alignment with limited resources. In ICCV, 2017.
 [6] Xingwei Cao and Qibin Zhao. Tensorizing generative adversarial nets. CoRR, abs/1710.10772, 2017.
 [7] Yunpeng Chen, Xiaojie Jin, Bingyi Kang, Jiashi Feng, and Shuicheng Yan. Sharing residual units through collective tensor factorization to improve deep neural networks. In Proceedings of the TwentySeventh International Joint Conference on Artificial Intelligence, IJCAI18, pages 635–641. International Joint Conferences on Artificial Intelligence Organization, 7 2018.
 [8] François Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv preprint, pages 1610–02357, 2017.
 [9] Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. CoRR, abs/1509.05009, 2015.
 [10] Matthieu Courbariaux, Yoshua Bengio, and JeanPierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NIPS, 2015.
 [11] Simon S Du and Jason D Lee. On the power of overparametrization in neural networks with quadratic activation. In ICML, 2018.
 [12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask rcnn. In ICCV, 2017.
 [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
 [14] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, 2017.
 [15] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
 [16] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017.
 [17] YongDeok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. CoRR, 05 2016.
 [18] Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM REVIEW, 51(3):455–500, 2009.
 [19] Jean Kossaifi, Aran Khanna, Zachary Lipton, Tommaso Furlanello, and Anima Anandkumar. Tensor contraction layers for parsimonious deep nets. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pages 1940–1946. IEEE, 2017.
 [20] Jean Kossaifi, Zachary C. Lipton, Aran Khanna, Tommaso Furlanello, and Anima Anandkumar. Tensor regression networks. CoRR, abs/1707.08308, 2018.
 [21] Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. Tensorly: Tensor learning in python. Journal of Machine Learning Research, 20(26):1–6, 2019.
 [22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
 [23] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky. Speedingup convolutional neural networks using finetuned cpdecomposition. CoRR, abs/1412.6553, 2014.
 [24] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.
 [25] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
 [26] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV, 2016.
 [27] Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry Vetrov. Tensorizing neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS’15, pages 442–450, 2015.
 [28] I. V. Oseledets. Tensortrain decomposition. SIAM J. Sci. Comput., 33(5):2295–2317, Sept. 2011.
 [29] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPSW, 2017.
 [30] Tomas Pfister, James Charles, and Andrew Zisserman. Flowing convnets for human pose estimation in videos. In ICCV, 2015.
 [31] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnornet: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
 [32] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster rcnn: Towards realtime object detection with region proposal networks. In NIPS, 2015.
 [33] Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic. 300 faces inthewild challenge: The first facial landmark localization challenge. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 397–403, 2013.
 [34] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largescale image recognition. arXiv, 2014.
 [35] Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the optimization landscape of overparameterized shallow neural networks. IEEE Transactions on Information Theory, 2018.
 [36] Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E. Convolutional neural networks with lowrank regularization. CoRR, abs/1511.06067, 2015.
 [37] Zhiqiang Tang, Xi Peng, Shijie Geng, Lingfei Wu, Shaoting Zhang, and Dimitris Metaxas. Quantized densely connected unets for efficient landmark localization. In ECCV, 2018.
 [38] Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2), 2012.
 [39] Jonathan J Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training of a convolutional network and a graphical model for human pose estimation. In NIPS, 2014.
 [40] Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In CVPR, 2014.
 [41] Frederick Tung and Greg Mori. Clipq: Deep network compression learning by inparallel pruningquantization. In CVPR, 2018.
 [42] ShihEn Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In CVPR, 2016.
 [43] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In CVPR, 2016.
 [44] Yongxin Yang and Timothy M. Hospedales. Deep multitask representation learning: A tensor factorisation approach. CoRR, abs/1605.06391, 2016.
 [45] Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. Longterm forecasting using tensortrain rnns. CoRR, abs/1711.00073, 2017.
 [46] Aojun Zhou, Anbang Yao, Kuan Wang, and Yurong Chen. Explicit losserroraware quantization for lowbit deep neural networks. In CVPR, 2018.