Abstract
Low-resource automatic speech recognition (ASR) is challenging, as thelow-resource target language data cannot well train an ASR model. To solve thisissue, meta-learning formulates ASR for each source language into many smallASR tasks and meta-learns a model initialization on all tasks from differentsource languages to access fast adaptation on unseen target languages. However,for different source languages, the quantity and difficulty vary greatlybecause of their different data scales and diverse phonological systems, whichleads to task-quantity and task-difficulty imbalance issues and thus a failureof multilingual meta-learning ASR (MML-ASR). In this work, we solve thisproblem by developing a novel adversarial meta sampling (AMS) approach toimprove MML-ASR. When sampling tasks in MML-ASR, AMS adaptively determines thetask sampling probability for each source language. Specifically, for eachsource language, if the query loss is large, it means that its tasks are notwell sampled to train ASR model in terms of its quantity and difficulty andthus should be sampled more frequently for extra learning. Inspired by thisfact, we feed the historical task query loss of all source language domain intoa network to learn a task sampling policy for adversarially increasing thecurrent query loss of MML-ASR. Thus, the learnt task sampling policy can masterthe learning situation of each language and thus predicts good task samplingprobability for each language for more effective learning. Finally, experimentresults on two multilingual datasets show significant performance improvementwhen applying our AMS on MML-ASR, and also demonstrate the applicability of AMSto other low-resource speech tasks and transfer learning ASR approaches.