Do convolutional networks really need a fixed feed-forward structure? Often,a neural network is already confident after a few layers about the high-levelconcept shown in the image. However, due to the fixed network structure, allremaining layers still need to be evaluated. What if the network could jumpright to a layer that is specialized in fine-grained differences of the image'scontent? In this work, we propose Adanets, a family of convolutional networkswith adaptive computation graphs. Following a high-level structure similar toresidual networks (Resnets), the key difference is that for each layer a gatingfunction determines whether to execute the layer or move on to the next one. Inexperiments on CIFAR-10 and ImageNet we demonstrate that Adanets efficientlyallocate computational budget among layers and learn distinct layersspecializing in similar categories. Adanet 50 achieves a top 5 error rate of7.94% on ImageNet using 30% fewer computations than Resnet 34, which onlyachieves 8.58%. Lastly, we study the effect of adaptive computation graphs onthe susceptibility towards adversarial examples. We observe that Adanets show ahigher robustness towards adversarial attacks, complementing other defensessuch as JPEG compression.