Abstract
This paper investigates different pretraining approaches to spoken languageidentification. The paper is based on our submission to the Oriental LanguageRecognition 2021 Challenge. We participated in two tracks of the challenge:constrained and unconstrained language recognition. For the constrained track,we first trained a Conformer-based encoder-decoder model for multilingualautomatic speech recognition (ASR), using the provided training data that hadtranscripts available. The shared encoder of the multilingual ASR model wasthen finetuned for the language identification task. For the unconstrainedtask, we relied on both externally available pretrained models as well asexternal data: the multilingual XLSR-53 wav2vec2.0 model was finetuned on theVoxLingua107 corpus for the language recognition task, and finally finetuned onthe provided target language training data, augmented with CommonVoice data.Our primary metric $C_{\rm avg}$ values on the Test set are 0.0079 for theconstrained task and 0.0119 for the unconstrained task which resulted in thesecond place in both rankings. In post-evaluation experiments, we study theamount of target language data needed for training an accurate backend model,the importance of multilingual pretraining data, and compare different modelsas finetuning starting points.