Abstract
The massive adoption of large language models (LLMs) demands efficientdeployment strategies. However, the auto-regressive decoding process, which isfundamental to how most LLMs generate text, poses challenges to achieveefficient serving. In this work, we introduce a parallel auto-regressivegeneration method. By instruct-tuning on general domain data that containshierarchical structures, we enable LLMs to independently plan their generationprocess and perform auto-parallel auto-regressive (APAR) generation,significantly reducing the number of generation steps. APAR alone can achieveup to 2x speed-up, and when combined with speculative decoding, the speed-upcan reach up to 4x. In addition, APAR reduces the key-value cache consumptionand attention computation during generation. This leads to a throughputincrease of 20-70% and a latency reduce of 20-35% in high-throughput scenarios,compared to state-of-the-art serving frameworks.