Abstract
Recent methods in speech and language technology pretrain very LARGE modelswhich are fine-tuned for specific tasks. However, the benefits of such LARGEmodels are often limited to a few resource rich languages of the world. In thiswork, we make multiple contributions towards building ASR systems for lowresource languages from the Indian subcontinent. First, we curate 17,000 hoursof raw speech data for 40 Indian languages from a wide variety of domainsincluding education, news, technology, and finance. Second, using this rawspeech data we pretrain several variants of wav2vec style models for 40 Indianlanguages. Third, we analyze the pretrained models to find key features:codebook vectors of similar sounding phonemes are shared across languages,representations across layers are discriminative of the language family, andattention heads often pay attention within small local windows. Fourth, wefine-tune this model for downstream ASR for 9 languages and obtainstate-of-the-art results on 3 public datasets, including on very low-resourcelanguages such as Sinhala and Nepali. Our work establishes that multilingualpretraining is an effective strategy for building ASR systems for thelinguistically diverse speakers of the Indian subcontinent.