Recent years have witnessed great success of convolutional neural network(CNN) for various problems both in low and high level visions. Especiallynoteworthy is the residual network which was originally proposed to handlehigh-level vision problems and enjoys several merits. This paper aims to extendthe merits of residual network, such as skip connection induced fast training,for a typical low-level vision problem, i.e., single image super-resolution. Ingeneral, the two main challenges of existing deep CNN for supper-resolution liein the gradient exploding/vanishing problem and large numbers of parameters orcomputational cost as CNN goes deeper. Correspondingly, the skip connections oridentity mapping shortcuts are utilized to avoid gradient exploding/vanishingproblem. In addition, the skip connections have naturally centered theactivation which led to better performance. To tackle with the second problem,a lightweight CNN architecture which has carefully designed width, depth andskip connections was proposed. In particular, a strategy of gradually varyingthe shape of network has been proposed for residual network. Different residualarchitectures for image super-resolution have also been compared. Experimentalresults have demonstrated that the proposed CNN model can not only achievestate-of-the-art PSNR and SSIM results for single image super-resolution butalso produce visually pleasant results. This paper has extended the mmm 2017oral conference paper with a considerable new analyses and more experimentsespecially from the perspective of centering activations and ensemble behaviorsof residual network.