Abstract
Teaching a multi-fingered dexterous robot to grasp objects in the real worldhas been a challenging problem due to its high dimensional state and actionspace. We propose a robot-learning system that can take a small number of humandemonstrations and learn to grasp unseen object poses given partially occludedobservations. Our system leverages a small motion capture dataset and generatesa large dataset with diverse and successful trajectories for a multi-fingeredrobot gripper. By adding domain randomization, we show that our datasetprovides robust grasping trajectories that can be transferred to a policylearner. We train a dexterous grasping policy that takes the point clouds ofthe object as input and predicts continuous actions to grasp objects fromdifferent initial robot states. We evaluate the effectiveness of our system ona 22-DoF floating Allegro Hand in simulation and a 23-DoF Allegro robot handwith a KUKA arm in real world. The policy learned from our dataset cangeneralize well on unseen object poses in both simulation and the real world