Abstract
Activation-aware low-rank factorization techniques yield strong compression results, but they are generally confined to linear layers, and existing whitening-based theory typically makes an implicit full-rank assumption on layer activations. We introduce a layer representation framework that allows us to understand and extend low-rank factorization beyond linear layers, including standard and grouped convolutions. Within this framework, we provide a broader characterization of whitening-based activation-aware factorization that naturally covers rank-deficient activations, and show that the resulting projection is optimal in terms of layer output distortion. We then use output distortion as a proxy to allocate per-layer ranks under explicit FLOP or parameter count budgets, with negligible overhead via a Lagrangian relaxation of the allocation problem. Together, these components form BALF, an end-to-end, fine-tuning-free pipeline for model compression. Across CNNs and vision transformers on CIFAR-10 and ImageNet-1K, BALF generally achieves higher accuracy than SVD-based factorization baselines at matched FLOP or parameter count targets, and remains competitive with other fine-tuning-free compression techniques.