Abstract
Despite the recent advancement in NLP research, cross-lingual transfer fornatural language generation is relatively understudied. In this work, wetransfer supervision from high resource language (HRL) to multiple low-resourcelanguages (LRLs) for natural language generation (NLG). We consider four NLGtasks (text summarization, question generation, news headline generation, anddistractor generation) and three syntactically diverse languages, i.e.,English, Hindi, and Japanese. We propose an unsupervised cross-lingual languagegeneration framework (called ZmBART) that does not use any parallel orpseudo-parallel/back-translated data. In this framework, we further pre-trainmBART sequence-to-sequence denoising auto-encoder model with an auxiliary taskusing monolingual data of three languages. The objective function of theauxiliary task is close to the target tasks which enriches the multi-linguallatent representation of mBART and provides good initialization for targettasks. Then, this model is fine-tuned with task-specific supervised Englishdata and directly evaluated with low-resource languages in the Zero-shotsetting. To overcome catastrophic forgetting and spurious correlation issues,we applied freezing model component and data argumentation approachesrespectively. This simple modeling approach gave us promising results.Weexperimented with few-shot training (with 1000 supervised data points) whichboosted the model performance further. We performed several ablations andcross-lingual transferability analyses to demonstrate the robustness of ZmBART.