Abstract
Sequence-to-sequence models are commonly trained via maximum likelihoodestimation (MLE). However, standard MLE training considers a word-levelobjective, predicting the next word given the previous ground-truth partialsentence. This procedure focuses on modeling local syntactic patterns, and mayfail to capture long-range semantic structure. We present a novel solution toalleviate these issues. Our approach imposes global sequence-level guidance vianew supervision based on optimal transport, enabling the overallcharacterization and preservation of semantic features. We further show thatthis method can be understood as a Wasserstein gradient flow trying to matchour model to the ground truth sequence distribution. Extensive experiments areconducted to validate the utility of the proposed approach, showing consistentimprovements over a wide variety of NLP tasks, including machine translation,abstractive text summarization, and image captioning.