Abstract
Ensembles of separate neural networks (NNs) have shown superior accuracy andconfidence calibration over single NN across tasks. Recent methods compressensembles within a single network via early exits or multi-input multi-outputframeworks. However, the landscape of these methods is fragmented thus far,making it difficult to choose the right approach for a given task. Furthermore,the algorithmic performance of these methods is behind the ensemble of separateNNs and requires extensive architecture tuning. We propose a novel methodologyunifying these approaches into a Single Architecture Ensemble (SAE). Our methodlearns the optimal number and depth of exits per ensemble input in a single NN.This enables the SAE framework to flexibly tailor its configuration for a givenarchitecture or application. We evaluate SAEs on image classification andregression across various network architecture types and sizes. We demonstratecompetitive accuracy or confidence calibration to baselines while reducing thecompute operations or parameter count by up to $1.5{\sim}3.7\times$.