We present a simple approach for text infilling, the task of predictingmissing spans of text at any position in a document. While infilling couldenable rich functionality especially for writing assistance tools, moreattention has been devoted to language modeling---a special case of infillingwhere text is predicted at the end of a document. In this paper, we aim toextend the capabilities of language models (LMs) to the more general task ofinfilling. To this end, we train (or fine-tune) off-the-shelf LMs on sequencescontaining the concatenation of artificially-masked text and the text which wasmasked. We show that this approach, which we call infilling by languagemodeling, can enable LMs to infill entire sentences effectively on threedifferent domains: short stories, scientific abstracts, and lyrics.Furthermore, we show that humans have difficulty identifying sentences infilledby our approach as machine-generated in the domain of short stories.