Abstract
This work presents methods for learning cross-lingual sentencerepresentations using paired or unpaired bilingual texts. We hypothesize thatthe cross-lingual alignment strategy is transferable, and therefore a modeltrained to align only two languages can encode multilingually more alignedrepresentations. And such transfer from bilingual alignment to multilingualalignment is a dual-pivot transfer from two pivot languages to other languagepairs. To study this theory, we train an unsupervised model with unpairedsentences and another single-pair supervised model with bitexts, both based onthe unsupervised language model XLM-R. The experiments evaluate the models asuniversal sentence encoders on the task of unsupervised bitext mining on twodatasets, where the unsupervised model reaches the state of the art ofunsupervised retrieval, and the alternative single-pair supervised modelapproaches the performance of multilingually supervised models. The resultssuggest that bilingual training techniques as proposed can be applied to getsentence representations with higher multilingual alignment.