Grasping is natural for humans. However, it involves complex handconfigurations and soft tissue deformation that can result in complicatedregions of contact between the hand and the object. Understanding and modelingthis contact can potentially improve hand models, AR/VR experiences, androbotic grasping. Yet, we currently lack datasets of hand-object contact pairedwith other data modalities, which is crucial for developing and evaluatingcontact modeling techniques. We introduce ContactPose, the first dataset ofhand-object contact paired with hand pose, object pose, and RGB-D images.ContactPose has 2306 unique grasps of 25 household objects grasped with 2functional intents by 50 participants, and more than 2.9 M RGB-D grasp images.Analysis of ContactPose data reveals interesting relationships between handpose and contact. We use this data to rigorously evaluate various datarepresentations, heuristics from the literature, and learning methods forcontact modeling. Data, code, and trained models are available athttps://contactpose.cc.gatech.edu.