Abstract
As large language models (LLMs) take on complex tasks, their inputs aresupplemented with longer contexts that incorporate domain knowledge oruser-specific information. Yet using long contexts poses a challenge forresponsive LLM systems, as nothing can be generated until the whole context isprocessed by the LLM. While the context-processing delay can be reduced byreusing the KV cache of a context across different inputs, fetching the KVcache, which contains large tensors, over the network can cause extra networkdelays. CacheGen is a fast context-loading module for LLM systems. First, CacheGenuses a custom tensor encoder, which embraces KV cache's distributionalproperties, to encode a KV cache into more compact bitstream representationswith negligible encoding/decoding overhead. This reduces the bandwidth demandto fetch the KV cache. Second, to maintain low context-loading delay and highgeneration quality, CacheGen adapts the streaming strategies to cope withchanges in available bandwidth. When available bandwidth drops, CacheGen mayraise the compression level for a part of the context or choose to recomputeits KV cache on the fly. We test CacheGen on four popular LLMs of various sizesand four datasets (662 contexts in total). Compared to the recent systems thatreuse the KV cache, CacheGen reduces the KV cache size by 3.7-4.3x and thetotal delay in fetching and processing contexts by 2.7-3.2x while havingnegligible impact on the LLM response quality in accuracy or perplexity.