The human ability to recognize objects is impaired when the object is notshown in full. "Minimal images" are the smallest regions of an image thatremain recognizable for humans. Ullman et al. 2016 show that a slightmodification of the location and size of the visible region of the minimalimage produces a sharp drop in human recognition accuracy. In this paper, wedemonstrate that such drops in accuracy due to changes of the visible regionare a common phenomenon between humans and existing state-of-the-art deepneural networks (DNNs), and are much more prominent in DNNs. We found manycases where DNNs classified one region correctly and the other incorrectly,though they only differed by one row or column of pixels, and were often biggerthan the average human minimal image size. We show that this phenomenon isindependent from previous works that have reported lack of invariance to minormodifications in object location in DNNs. Our results thus reveal a new failuremode of DNNs that also affects humans to a much lesser degree. They expose howfragile DNN recognition ability is for natural images even without adversarialpatterns being introduced. Bringing the robustness of DNNs in natural images tothe human level remains an open challenge for the community.