Abstract
Multilingual neural machine translation models generally distinguishtranslation directions by the language tag (LT) in front of the source ortarget sentences. However, current LT strategies cannot indicate the desiredtarget language as expected on zero-shot translation, i.e., the off-targetissue. Our analysis reveals that the indication of the target language issensitive to the placement of the target LT. For example, when placing thetarget LT on the decoder side, the indication would rapidly degrade along withdecoding steps, while placing the target LT on the encoder side would lead tocopying or paraphrasing the source input. To address the above issues, wepropose a simple yet effective strategy named Language Converter Strategy(LCS). By introducing the target language embedding into the top encoderlayers, LCS mitigates confusion in the encoder and ensures stable languageindication for the decoder. Experimental results on MultiUN, TED, and OPUS-100datasets demonstrate that LCS could significantly mitigate the off-targetissue, with language accuracy up to 95.28%, 96.21%, and 85.35% meanwhileoutperforming the vanilla LT strategy by 3.07, 3,3, and 7.93 BLEU scores onzero-shot translation, respectively.