Training a model for grammatical error correction (GEC) requires a set oflabeled ungrammatical / grammatical sentence pairs, but manually annotatingsuch pairs can be expensive. Recently, the Break-It-Fix-It (BIFI) framework hasdemonstrated strong results on learning to repair a broken program without anylabeled examples, but this relies on a perfect critic (e.g., a compiler) thatreturns whether an example is valid or not, which does not exist for the GECtask. In this work, we show how to leverage a pretrained language model (LM) indefining an LM-Critic, which judges a sentence to be grammatical if the LMassigns it a higher probability than its local perturbations. We apply thisLM-Critic and BIFI along with a large set of unlabeled sentences to bootstraprealistic ungrammatical / grammatical pairs for training a corrector. Weevaluate our approach on GEC datasets across multiple domains (CoNLL-2014,BEA-2019, GMEG-wiki and GMEG-yahoo) and show that it outperforms existingmethods in both the unsupervised setting (+7.7 F0.5) and the supervised setting(+0.5 F0.5).