Abstract
What does it even mean to average neural networks? We investigate the problemof synthesizing a single neural network from a collection of pretrained models,each trained on disjoint data shards, using only their final weights and noaccess to training data. In forming a definition of neural averaging, we takeinsight from model soup, which appears to aggregate multiple models into asingular model while enhancing generalization performance. In this work, wereinterpret model souping as a special case of a broader framework: AmortizedModel Ensembling (AME) for neural averaging, a data-free meta-optimizationapproach that treats model differences as pseudogradients to guide neuralweight updates. We show that this perspective not only recovers model soup butenables more expressive and adaptive ensembling strategies. Empirically, AMEproduces averaged neural solutions that outperform both individual experts andmodel soup baselines, especially in out-of-distribution settings. Our resultssuggest a principled and generalizable notion of data-free model weightaggregation and defines, in one sense, how to perform neural averaging.