Abstract
Recent works show we can linearize large language models (LLMs) -- swappingthe quadratic attentions of popular Transformer-based LLMs with subquadraticanalogs, such as linear attention -- avoiding the expensive pretraining costs.However, linearizing LLMs often significantly degrades model quality, stillrequires training over billions of tokens, and remains limited to smaller 1.3Bto 7B LLMs. We thus propose Low-rank Linear Conversion via Attention Transfer(LoLCATs), a simple two-step method that improves LLM linearizing quality withorders of magnitudes less memory and compute. We base these steps on twofindings. First, we can replace an LLM's softmax attentions withclosely-approximating linear attentions, simply by training the linearattentions to match their softmax counterparts with an output MSE loss("attention transfer"). Then, this enables adjusting for approximation errorsand recovering LLM quality simply with low-rank adaptation (LoRA). LoLCATssignificantly improves linearizing quality, training efficiency, andscalability. We significantly reduce the linearizing quality gap and producestate-of-the-art subquadratic LLMs from Llama 3 8B and Mistral 7B v0.1, leadingto 20+ points of improvement on 5-shot MMLU. Furthermore, LoLCATs does so withonly 0.2% of past methods' model parameters and 0.4% of their training tokens.Finally, we apply LoLCATs to create the first linearized 70B and 405B LLMs (50xlarger than prior work). When compared with prior approaches under the samecompute budgets, LoLCATs significantly improves linearizing quality, closingthe gap between linearized and original Llama 3.1 70B and 405B LLMs by 77.8%and 78.1% on 5-shot MMLU.