Abstract
Controllable Text Generation (CTG) is a vital subfield in Natural LanguageProcessing (NLP), aiming to generate text that aligns with desired attributes.However, previous studies commonly focus on the quality of controllable textgeneration for short sequences, while the generation of long-form text remainslargely underexplored. In this paper, we observe that the controllability oftexts generated by the powerful prefix-based method Air-Decoding tends todecline with increasing sequence length, which we hypothesize primarily arisesfrom the observed decay in attention to the prefixes. Meanwhile, differenttypes of prefixes including soft and hard prefixes are also key factorsinfluencing performance. Building on these insights, we propose a lightweightand effective framework called Dynamic Token-level Prefix Augmentation (DTPA)based on Air-Decoding for controllable text generation. Specifically, it firstselects the optimal prefix type for a given task. Then we dynamically amplifythe attention to the prefix for the attribute distribution to enhancecontrollability, with a scaling factor growing exponentially as the sequencelength increases. Moreover, based on the task, we optionally apply a similaraugmentation to the original prompt for the raw distribution to balance textquality. After attribute distribution reconstruction, the generated textsatisfies the attribute constraints well. Experiments on multiple CTG tasksdemonstrate that DTPA generally outperforms other methods in attribute controlwhile maintaining competitive fluency, diversity, and topic relevance. Furtheranalysis highlights DTPA's superior effectiveness in long text generation.