Text generation has made significant advances in the last few years. Yet,evaluation metrics have lagged behind, as the most popular choices (e.g., BLEUand ROUGE) may correlate poorly with human judgments. We propose BLEURT, alearned evaluation metric based on BERT that can model human judgments with afew thousand possibly biased training examples. A key aspect of our approach isa novel pre-training scheme that uses millions of synthetic examples to helpthe model generalize. BLEURT provides state-of-the-art results on the lastthree years of the WMT Metrics shared task and the WebNLG Competition dataset.In contrast to a vanilla BERT-based approach, it yields superior results evenwhen the training data is scarce and out-of-distribution.