An efficient, generalizable physical simulator with universal uncertaintyestimates has wide applications in robot state estimation, planning, andcontrol. In this paper, we build such a simulator for two scenarios, planarpushing and ball bouncing, by augmenting an analytical rigid-body simulatorwith a neural network that learns to model uncertainty as residuals. Combiningsymbolic, deterministic simulators with learnable, stochastic neural netsprovides us with expressiveness, efficiency, and generalizabilitysimultaneously. Our model outperforms both purely analytical and purely learnedsimulators consistently on real, standard benchmarks. Compared with methodsthat model uncertainty using Gaussian processes, our model runs much faster,generalizes better to new object shapes, and is able to characterize thecomplex distribution of object trajectories.