Abstract
While recent benchmarks have spurred a lot of new work on improving thegeneralization of pretrained multilingual language models on multilingualtasks, techniques to improve code-switched natural language understanding taskshave been far less explored. In this work, we propose the use of bilingualintermediate pretraining as a reliable technique to derive large and consistentperformance gains on three different NLP tasks using code-switched text. Weachieve substantial absolute improvements of 7.87%, 20.15%, and 10.99%, on themean accuracies and F1 scores over previous state-of-the-art systems forHindi-English Natural Language Inference (NLI), Question Answering (QA) tasks,and Spanish-English Sentiment Analysis (SA) respectively. We show consistentperformance gains on four different code-switched language-pairs(Hindi-English, Spanish-English, Tamil-English and Malayalam-English) for SA.We also present a code-switched masked language modelling (MLM) pretrainingtechnique that consistently benefits SA compared to standard MLM pretrainingusing real code-switched text.