Abstract
Conventional notions of generalization often fail to describe the ability oflearned models to capture meaningful information from dynamical data. A neuralnetwork that learns complex dynamics with a small test error may still fail toreproduce its \emph{physical} behavior, including associated statisticalmoments and Lyapunov exponents. To address this gap, we propose an ergodictheoretic approach to generalization of complex dynamical models learned fromtime series data. Our main contribution is to define and analyze generalizationof a broad suite of neural representations of classes of ergodic systems,including chaotic systems, in a way that captures emulating underlyinginvariant, physical measures. Our results provide theoretical justification forwhy regression methods for generators of dynamical systems (Neural ODEs) failto generalize, and why their statistical accuracy improves upon adding Jacobianinformation during training. We verify our results on a number of ergodicchaotic systems and neural network parameterizations, including MLPs, ResNets,Fourier Neural layers, and RNNs.