Abstract
This work explores the potential of using differentiable simulation forlearning quadruped locomotion. Differentiable simulation promises fastconvergence and stable training by computing low-variance first-order gradientsusing robot dynamics. However, its usage for legged robots is still limited tosimulation. The main challenge lies in the complex optimization landscape ofrobotic tasks due to discontinuous dynamics. This work proposes a newdifferentiable simulation framework to overcome these challenges. Our approachcombines a high-fidelity, non-differentiable simulator for forward dynamicswith a simplified surrogate model for gradient backpropagation. This approachmaintains simulation accuracy by aligning the robot states from the surrogatemodel with those of the precise, non-differentiable simulator. Our frameworkenables learning quadruped walking in simulation in minutes withoutparallelization. When augmented with GPU parallelization, our approach allowsthe quadruped robot to master diverse locomotion skills on challenging terrainsin minutes. We demonstrate that differentiable simulation outperforms areinforcement learning algorithm (PPO) by achieving significantly better sampleefficiency while maintaining its effectiveness in handling large-scaleenvironments. Our method represents one of the first successful applications ofdifferentiable simulation to real-world quadruped locomotion, offering acompelling alternative to traditional RL methods.