Abstract
Neural machine translation usually adopts autoregressive models and suffersfrom exposure bias as well as the consequent error propagation problem. Manyprevious works have discussed the relationship between error propagation andthe \emph{accuracy drop} (i.e., the left part of the translated sentence isoften better than its right part in left-to-right decoding models) problem. Inthis paper, we conduct a series of analyses to deeply understand this problemand get several interesting findings. (1) The role of error propagation onaccuracy drop is overstated in the literature, although it indeed contributesto the accuracy drop problem. (2) Characteristics of a language play a moreimportant role in causing the accuracy drop: the left part of the translationresult in a right-branching language (e.g., English) is more likely to be moreaccurate than its right part, while the right part is more accurate for aleft-branching language (e.g., Japanese). Our discoveries are confirmed ondifferent model structures including Transformer and RNN, and in other sequencegeneration tasks such as text summarization.