Perceiving accurate 3D object shape is important for robots to interact withthe physical world. Current research along this direction has been primarilyrelying on visual observations. Vision, however useful, has inherentlimitations due to occlusions and the 2D-3D ambiguities, especially forperception with a monocular camera. In contrast, touch gets precise local shapeinformation, though its efficiency for reconstructing the entire shape could below. In this paper, we propose a novel paradigm that efficiently perceivesaccurate 3D object shape by incorporating visual and tactile observations, aswell as prior knowledge of common object shapes learned from large-scale shaperepositories. We use vision first, applying neural networks with learned shapepriors to predict an object's 3D shape from a single-view color image. We thenuse tactile sensing to refine the shape; the robot actively touches the objectregions where the visual prediction has high uncertainty. Our methodefficiently builds the 3D shape of common objects from a color image and asmall number of tactile explorations (around 10). Our setup is easy to applyand has potentials to help robots better perform grasping or manipulation taskson real-world objects.