Abstract
Autoregressive generative models naturally generate variable-lengthsequences, while non-autoregressive models struggle, often imposing rigid,token-wise structures. We propose Edit Flows, a non-autoregressive model thatovercomes these limitations by defining a discrete flow over sequences throughedit operations$\unicode{x2013}$insertions, deletions, and substitutions. Bymodeling these operations within a Continuous-time Markov Chain over thesequence space, Edit Flows enable flexible, position-relative generation thataligns more closely with the structure of sequence data. Our training methodleverages an expanded state space with auxiliary variables, making the learningprocess efficient and tractable. Empirical results show that Edit Flowsoutperforms both autoregressive and mask models on image captioning andsignificantly outperforms the mask construction in text and code generation.