Variational inference (VI) has become the method of choice for fitting manymodern probabilistic models. However, practitioners are faced with a fragmentedliterature that offers a bewildering array of algorithmic options. First, thevariational family. Second, the granularity of the updates e.g. whether theupdates are local to each data point and employ message passing or global.Third, the method of optimization (bespoke or blackbox, closed-form orstochastic updates, etc.). This paper presents a new framework, termedPartitioned Variational Inference (PVI), that explicitly acknowledges thesealgorithmic dimensions of VI, unifies disparate literature, and providesguidance on usage. Crucially, the proposed PVI framework allows us to identifynew ways of performing VI that are ideally suited to challenging learningscenarios including federated learning (where distributed computing isleveraged to process non-centralized data) and continual learning (where newdata and tasks arrive over time and must be accommodated quickly). We showcasethese new capabilities by developing communication-efficient federated trainingof Bayesian neural networks and continual learning for Gaussian process modelswith private pseudo-points. The new methods significantly outperform thestate-of-the-art, whilst being almost as straightforward to implement asstandard VI.