Abstract
Normalizing flows and autoregressive models have been successfully combinedto produce state-of-the-art results in density estimation, via MaskedAutoregressive Flows (MAF), and to accelerate state-of-the-art WaveNet-basedspeech synthesis to 20x faster than real-time, via Inverse Autoregressive Flows(IAF). We unify and generalize these approaches, replacing the (conditionally)affine univariate transformations of MAF/IAF with a more general class ofinvertible univariate transformations expressed as monotonic neural networks.We demonstrate that the proposed neural autoregressive flows (NAF) areuniversal approximators for continuous probability distributions, and theirgreater expressivity allows them to better capture multimodal targetdistributions. Experimentally, NAF yields state-of-the-art performance on asuite of density estimation tasks and outperforms IAF in variationalautoencoders trained on binarized MNIST.