Abstract
Multilingual speech recognition with neural networks is often implementedwith batch-learning, when all of the languages are available before training.An ability to add new languages after the prior training sessions can beeconomically beneficial, but the main challenge is catastrophic forgetting. Inthis work, we combine the qualities of weight factorization and elastic weightconsolidation in order to counter catastrophic forgetting and facilitatelearning new languages quickly. Such combination allowed us to eliminatecatastrophic forgetting while still achieving performance for the new languagescomparable with having all languages at once, in experiments of learning froman initial 10 languages to achieve 26 languages without catastrophic forgettingand a reasonable performance compared to training all languages from scratch.