Abstract
Generating high-quality text with sufficient diversity is essential for awide range of Natural Language Generation (NLG) tasks. Maximum-Likelihood (MLE)models trained with teacher forcing have constantly been reported as weakbaselines, where poor performance is attributed to exposure bias; at inferencetime, the model is fed its own prediction instead of a ground-truth token,which can lead to accumulating errors and poor samples. This line of reasoninghas led to an outbreak of adversarial based approaches for NLG, on the accountthat GANs do not suffer from exposure bias. In this work, wake make severalsurprising observations with contradict common beliefs. We first revisit thecanonical evaluation framework for NLG, and point out fundamental flaws withquality-only evaluation: we show that one can outperform such metrics using asimple, well-known temperature parameter to artificially reduce the entropy ofthe model's conditional distributions. Second, we leverage the control over thequality / diversity tradeoff given by this parameter to evaluate models overthe whole quality-diversity spectrum, and find MLE models constantly outperformthe proposed GAN variants, over the whole quality-diversity space. Our resultshave several implications: 1) The impact of exposure bias on sample quality isless severe than previously thought, 2) temperature tuning provides a betterquality / diversity trade off than adversarial training, while being easier totrain, easier to cross-validate, and less computationally expensive.