We propose Blank Language Model (BLM), a model that generates sequences bydynamically creating and filling in blanks. Unlike previous masked languagemodels or the Insertion Transformer, BLM uses blanks to control which part ofthe sequence to expand. This fine-grained control of generation is ideal for avariety of text editing and rewriting tasks. The model can start from a singleblank or partially completed text with blanks at specified locations. Ititeratively determines which word to place in a blank and whether to insert newblanks, and stops generating when no blanks are left to fill. BLM can beefficiently trained using a lower bound of the marginal data likelihood, andachieves perplexity comparable to traditional left-to-right language models onthe Penn Treebank and WikiText datasets. On the task of filling missing textsnippets, BLM significantly outperforms all other baselines in terms of bothaccuracy and fluency. Experiments on style transfer and damaged ancient textrestoration demonstrate the potential of this framework for a wide range ofapplications.