Abstract
We address the challenge of developing a generalizable neural trackingcontroller for dexterous manipulation from human references. This controlleraims to manage a dexterous robot hand to manipulate diverse objects for variouspurposes defined by kinematic human-object interactions. Developing such acontroller is complicated by the intricate contact dynamics of dexterousmanipulation and the need for adaptivity, generalizability, and robustness.Current reinforcement learning and trajectory optimization methods often fallshort due to their dependence on task-specific rewards or precise systemmodels. We introduce an approach that curates large-scale successful robottracking demonstrations, comprising pairs of human references and robotactions, to train a neural controller. Utilizing a data flywheel, weiteratively enhance the controller's performance, as well as the number andquality of successful tracking demonstrations. We exploit available trackingdemonstrations and carefully integrate reinforcement learning and imitationlearning to boost the controller's performance in dynamic environments. At thesame time, to obtain high-quality tracking demonstrations, we individuallyoptimize per-trajectory tracking by leveraging the learned tracking controllerin a homotopy optimization method. The homotopy optimization, mimickingchain-of-thought, aids in solving challenging trajectory tracking problems toincrease demonstration diversity. We showcase our success by training ageneralizable neural controller and evaluating it in both simulation and realworld. Our method achieves over a 10% improvement in success rates compared toleading baselines. The project website with animated results is available athttps://meowuu7.github.io/DexTrack/.