Abstract
We introduce a novel, end-to-end learnable, differentiable non-rigid trackerthat enables state-of-the-art non-rigid reconstruction by a learned robustoptimization. Given two input RGB-D frames of a non-rigidly moving object, weemploy a convolutional neural network to predict dense correspondences andtheir confidences. These correspondences are used as constraints in anas-rigid-as-possible (ARAP) optimization problem. By enabling gradientback-propagation through the weighted non-linear least squares solver, we areable to learn correspondences and confidences in an end-to-end manner such thatthey are optimal for the task of non-rigid tracking. Under this formulation,correspondence confidences can be learned via self-supervision, informing alearned robust optimization, where outliers and wrong correspondences areautomatically down-weighted to enable effective tracking. Compared tostate-of-the-art approaches, our algorithm shows improved reconstructionperformance, while simultaneously achieving 85 times faster correspondenceprediction than comparable deep-learning based methods. We make our codeavailable.