Abstract
We show that Normalising Flows become pathological when used to model targetswhose supports have complicated topologies. In this scenario, we prove that aflow must become arbitrarily numerically noninvertible in order to approximatethe target closely. This result has implications for all flow-based models, andespecially Residual Flows (ResFlows), which explicitly control the Lipschitzconstant of the bijection used. To address this, we propose ContinuouslyIndexed Flows (CIFs), which replace the single bijection used by normalisingflows with a continuously indexed family of bijections, and which canintuitively "clean up" mass that would otherwise be misplaced by a singlebijection. We show that CIFs can exactly match the support of a target evenwhen its topology differs from the prior, and obtain empirically betterperformance for a variety of models and benchmarks.