Abstract
End-to-end autonomous driving (E2EAD) systems, which learn to predict futuretrajectories directly from sensor data, are fundamentally challenged by theinherent spatio-temporal imbalance of trajectory data. This imbalance creates asignificant optimization burden, causing models to learn spurious correlationsinstead of causal inference, while also prioritizing uncertain, distantpredictions, thereby compromising immediate safety. To address these issues, wepropose ResAD, a novel Normalized Residual Trajectory Modeling framework.Instead of predicting the future trajectory directly, our approach reframes thelearning task to predict the residual deviation from a deterministic inertialreference. The inertial reference serves as a counterfactual, forcing the modelto move beyond simple pattern recognition and instead identify the underlyingcausal factors (e.g., traffic rules, obstacles) that necessitate deviationsfrom a default, inertially-guided path. To deal with the optimization imbalancecaused by uncertain, long-term horizons, ResAD further incorporates Point-wiseNormalization of the predicted residual. It re-weights the optimizationobjective, preventing large-magnitude errors associated with distant, uncertainwaypoints from dominating the learning signal. Extensive experiments validatethe effectiveness of our framework. On the NAVSIM benchmark, ResAD achieves astate-of-the-art PDMS of 88.6 using a vanilla diffusion policy with only twodenoising steps, demonstrating that our approach significantly simplifies thelearning task and improves model performance. The code will be released tofacilitate further research.