Abstract
We introduce Dynamic Deep Neural Networks (D2NN), a new type of feed-forwarddeep neural network that allows selective execution. Given an input, only asubset of D2NN neurons are executed, and the particular subset is determined bythe D2NN itself. By pruning unnecessary computation depending on input, D2NNsprovide a way to improve computational efficiency. To achieve dynamic selectiveexecution, a D2NN augments a feed-forward deep neural network (directed acyclicgraph of differentiable modules) with controller modules. Each controllermodule is a sub-network whose output is a decision that controls whether othermodules can execute. A D2NN is trained end to end. Both regular and controllermodules in a D2NN are learnable and are jointly trained to optimize bothaccuracy and efficiency. Such training is achieved by integratingbackpropagation with reinforcement learning. With extensive experiments ofvarious D2NN architectures on image classification tasks, we demonstrate thatD2NNs are general and flexible, and can effectively optimizeaccuracy-efficiency trade-offs.