Abstract
Autoregressive generative models naturally generate variable-lengthsequences, while non-autoregressive models struggle, often imposing rigid,token-wise structures. We propose Edit Flows, a non-autoregressive model thatovercomes these limitations by defining a discrete flow over sequences throughedit operations-insertions, deletions, and substitutions. By modeling theseoperations within a Continuous-time Markov Chain over the sequence space, EditFlows enable flexible, position-relative generation that aligns more closelywith the structure of sequence data. Our training method leverages an expandedstate space with auxiliary variables, making the learning process efficient andtractable. Empirical results show that Edit Flows outperforms bothautoregressive and mask models on image captioning and significantlyoutperforms the mask construction in text and code generation.