Abstract
Is there really much more to say about sparse autoencoders (SAEs)?Autoencoders in general, and SAEs in particular, represent deep architecturesthat are capable of modeling low-dimensional latent structure in data. Suchstructure could reflect, among other things, correlation patterns in largelanguage model activations, or complex natural image manifolds. And yet despitethe wide-ranging applicability, there have been relatively few changes to SAEsbeyond the original recipe from decades ago, namely, standard deepencoder/decoder layers trained with a classical/deterministic sparseregularizer applied within the latent space. One possible exception is thevariational autoencoder (VAE), which adopts a stochastic encoder module capableof producing sparse representations when applied to manifold data. In this workwe formalize underappreciated weaknesses with both canonical SAEs, as well asanalogous VAEs applied to similar tasks, and propose a hybrid alternative modelthat circumvents these prior limitations. In terms of theoretical support, weprove that global minima of our proposed model recover certain forms ofstructured data spread across a union of manifolds. Meanwhile, empiricalevaluations on synthetic and real-world datasets substantiate the efficacy ofour approach in accurately estimating underlying manifold dimensions andproducing sparser latent representations without compromising reconstructionerror. In general, we are able to exceed the performance of equivalent-capacitySAEs and VAEs, as well as recent diffusion models where applicable, withindomains such as images and language model activation patterns.