Abstract
Despite excellent performance on stationary test sets, deep neural networks(DNNs) can fail to generalize to out-of-distribution (OoD) inputs, includingnatural, non-adversarial ones, which are common in real-world settings. In thispaper, we present a framework for discovering DNN failures that harnesses 3Drenderers and 3D models. That is, we estimate the parameters of a 3D rendererthat cause a target DNN to misbehave in response to the rendered image. Usingour framework and a self-assembled dataset of 3D objects, we investigate thevulnerability of DNNs to OoD poses of well-known objects in ImageNet. Forobjects that are readily recognized by DNNs in their canonical poses, DNNsincorrectly classify 97% of their pose space. In addition, DNNs are highlysensitive to slight pose perturbations. Importantly, adversarial poses transferacross models and datasets. We find that 99.9% and 99.4% of the posesmisclassified by Inception-v3 also transfer to the AlexNet and ResNet-50 imageclassifiers trained on the same ImageNet dataset, respectively, and 75.5%transfer to the YOLOv3 object detector trained on MS COCO.