Abstract
Large Language Models (LLMs) have exhibited exceptional performance across aspectrum of natural language processing tasks. However, their substantial sizespose considerable challenges, particularly in computational demands andinference speed, due to their quadratic complexity. In this work, we haveidentified a key pattern: certain seemingly meaningless special tokens (i.e.,separators) contribute disproportionately to attention scores compared tosemantically meaningful tokens. This observation suggests that information ofthe segments between these separator tokens can be effectively condensed intothe separator tokens themselves without significant information loss. Guided bythis insight, we introduce SepLLM, a plug-and-play framework that acceleratesinference by compressing these segments and eliminating redundant tokens.Additionally, we implement efficient kernels for training acceleration.Experimental results across training-free, training-from-scratch, andpost-training settings demonstrate SepLLM's effectiveness. Notably, using theLlama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on theGSM8K-CoT benchmark while maintaining comparable performance. Furthermore, instreaming settings, SepLLM effectively processes sequences of up to 4 milliontokens or more while maintaining consistent language modeling capabilities.