Abstract
Rapid progress in Neural Machine Translation (NMT) systems over the last fewyears has been driven primarily towards improving translation quality, and as asecondary focus, improved robustness to input perturbations (e.g. spelling andgrammatical mistakes). While performance and robustness are importantobjectives, by over-focusing on these, we risk overlooking other importantproperties. In this paper, we draw attention to the fact that for someapplications, faithfulness to the original (input) text is important topreserve, even if it means introducing unusual language patterns in the(output) translation. We propose a simple, novel way to quantify whether an NMTsystem exhibits robustness and faithfulness, focusing on the case of word-orderperturbations. We explore a suite of functions to perturb the word order ofsource sentences without deleting or injecting tokens, and measure the effectson the target side in terms of both robustness and faithfulness. Across severalexperimental conditions, we observe a strong tendency towards robustness ratherthan faithfulness. These results allow us to better understand the trade-offbetween faithfulness and robustness in NMT, and opens up the possibility ofdeveloping systems where users have more autonomy and control in selectingwhich property is best suited for their use case.