Abstract
We propose data and knowledge-driven approaches for multilingual training ofthe automated speech recognition (ASR) system for a target language by poolingspeech data from multiple source languages. Exploiting the acousticsimilarities between Indian languages, we implement two approaches. Inphone/senone mapping, deep neural network (DNN) learns to map senones or phonesfrom one language to the others, and the transcriptions of the source languagesare modified such that they can be used along with the target language data totrain and fine-tune the target language ASR system. In the other approach, wemodel the acoustic information for all the languages simultaneously by traininga multitask DNN (MTDNN) to predict the senones of each language in differentoutput layers. The cross-entropy loss and the weight update procedure aremodified such that only the shared layers and the output layer responsible forpredicting the senone classes of a language are updated during training, if thefeature vector belongs to that particular language. In the low-resource setting(LRS), 40 hours of transcribed data each for Tamil, Telugu and Gujaratilanguages are used for training. The DNN based senone mapping technique givesrelative improvements in word error rates (WER) of 9.66%, 7.2% and 15.21% overthe baseline system for Tamil, Gujarati and Telugu languages, respectively. Inmedium-resourced setting (MRS), 160, 275 and 135 hours of data for Tamil,Kannada and Hindi languages are used, where, the same technique gives betterrelative improvements of 13.94%, 10.28% and 27.24% for Tamil, Kannada andHindi, respectively. The MTDNN with senone mapping based training in LRS, giveshigher relative WER improvements of 15.0%, 17.54% and 16.06%, respectively forTamil, Gujarati and Telugu, whereas in MRS, we see improvements of 21.24%21.05% and 30.17% for Tamil, Kannada and Hindi languages, respectively.