Abstract
Recent years have seen remarkable progress of text generation in differentcontexts, such as the most common setting of generating text from scratch, andthe emerging paradigm of retrieval-and-rewriting. Text infilling, which fillsmissing text portions of a sentence or paragraph, is also of numerous use inreal life, yet is under-explored. Previous work has focused on restrictedsettings by either assuming single word per missing portion or limiting to asingle missing portion to the end of the text. This paper studies the generaltask of text infilling, where the input text can have an arbitrary number ofportions to be filled, each of which may require an arbitrary unknown number oftokens. We study various approaches for the task, including a self-attentionmodel with segment-aware position encoding and bidirectional context modeling.We create extensive supervised data by masking out text with varyingstrategies. Experiments show the self-attention model greatly outperformsothers, creating a strong baseline for future research.