Pruning large neural networks while maintaining the performance is oftenhighly desirable due to the reduced space and time complexity. In existingmethods, pruning is incorporated within an iterative optimization procedurewith either heuristically designed pruning schedules or additionalhyperparameters, undermining their utility. In this work, we present a newapproach that prunes a given network once at initialization. Specifically, weintroduce a saliency criterion based on connection sensitivity that identifiesstructurally important connections in the network for the given task evenbefore training. This eliminates the need for both pretraining as well as thecomplex pruning schedule while making it robust to architecture variations.After pruning, the sparse network is trained in the standard way. Our methodobtains extremely sparse networks with virtually the same accuracy as thereference network on image classification tasks and is broadly applicable tovarious architectures including convolutional, residual and recurrent networks.Unlike existing methods, our approach enables us to demonstrate that theretained connections are indeed relevant to the given task.