USM-SCD: Multilingual Speaker Change Detection Based on Large Pretrained Foundation Models

  • 2023-09-14 21:46:49
  • Guanlong Zhao, Yongqiang Wang, Jason Pelecanos, Yu Zhang, Hank Liao, Yiling Huang, Han Lu, Quan Wang
  • 0


We introduce a multilingual speaker change detection model (USM-SCD) that cansimultaneously detect speaker turns and perform ASR for 96 languages. Thismodel is adapted from a speech foundation model trained on a large quantity ofsupervised and unsupervised data, demonstrating the utility of fine-tuning froma large generic foundation model for a downstream task. We analyze theperformance of this multilingual speaker change detection model through aseries of ablation studies. We show that the USM-SCD model can achieve morethan 75% average speaker change detection F1 score across a test set thatconsists of data from 96 languages. On American English, the USM-SCD model canachieve an 85.8% speaker change detection F1 score across various public andinternal test sets, beating the previous monolingual baseline model by 21%relative. We also show that we only need to fine-tune one-quarter of thetrainable model parameters to achieve the best model performance. The USM-SCDmodel exhibits state-of-the-art ASR quality compared with a strong public ASRbaseline, making it suitable to handle both tasks with negligible additionalcomputational cost.


Quick Read (beta)

loading the full paper ...