Abstract
Pre-trained language models have recently been shown to be able to performtranslation without finetuning via prompting. Inspired by these findings, westudy improving the performance of pre-trained language models on translationtasks, where training neural machine translation models is the current de factoapproach. We present Multi-Stage Prompting, a simple and lightweight approachfor better adapting pre-trained language models to translation tasks. To makepre-trained language models better translators, we divide the translationprocess via pre-trained language models into three separate stages: theencoding stage, the re-encoding stage, and the decoding stage. During eachstage, we independently apply different continuous prompts for allowingpre-trained language models better adapting to translation tasks. We conductextensive experiments on low-, medium-, and high-resource translation tasks.Experiments show that our method can significantly improve the translationperformance of pre-trained language models.