Classical deep convolutional networks increase receptive field size by eithergradual resolution reduction or application of hand-crafted dilatedconvolutions to prevent increase in the number of parameters. In this paper wepropose a novel displaced aggregation unit (DAU) that does not requirehand-crafting. In contrast to classical filters with units (pixels) placed on afixed regular grid, the displacement of the DAUs are learned, which enablesfilters to spatially-adapt their receptive field to a given problem. Weextensively demonstrate the strength of DAUs on a classification and semanticsegmentation tasks. Compared to ConvNets with regular filter, ConvNets withDAUs achieve comparable performance at faster convergence and up to 3-timesreduction in parameters. Furthermore, DAUs allow us to study deep networks fromnovel perspectives. We study spatial distributions of DAU filters and analyzethe number of parameters allocated for spatial coverage in a filter.