Abstract
Expert demonstrations are a rich source of supervision for training visualrobotic manipulation policies, but imitation learning methods often requireeither a large number of demonstrations or expensive online expert supervisionto learn reactive closed-loop behaviors. In this work, we introduce SPARTN(Synthetic Perturbations for Augmenting Robot Trajectories via NeRF): afully-offline data augmentation scheme for improving robot policies that useeye-in-hand cameras. Our approach leverages neural radiance fields (NeRFs) tosynthetically inject corrective noise into visual demonstrations, using NeRFsto generate perturbed viewpoints while simultaneously calculating thecorrective actions. This requires no additional expert supervision orenvironment interaction, and distills the geometric information in NeRFs into areal-time reactive RGB-only policy. In a simulated 6-DoF visual graspingbenchmark, SPARTN improves success rates by 2.8$\times$ over imitation learningwithout the corrective augmentations and even outperforms some methods that useonline supervision. It additionally closes the gap between RGB-only and RGB-Dsuccess rates, eliminating the previous need for depth sensors. In real-world6-DoF robotic grasping experiments from limited human demonstrations, ourmethod improves absolute success rates by $22.5\%$ on average, includingobjects that are traditionally challenging for depth-based methods. See videoresults at \url{https://bland.website/spartn}.