Multilingual Gloss-free Sign Language Translation: Towards Building a Sign Language Foundation Model

  • 2025-05-30 09:47:44
  • Sihan Tan, Taro Miyazaki, Kazuhiro Nakadai
  • 0

Abstract

Sign Language Translation (SLT) aims to convert sign language (SL) videosinto spoken language text, thereby bridging the communication gap between thesign and the spoken community. While most existing works focus on translating asingle sign language into a single spoken language (one-to-one SLT), leveragingmultilingual resources could mitigate low-resource issues and enhanceaccessibility. However, multilingual SLT (MLSLT) remains unexplored due tolanguage conflicts and alignment difficulties across SLs and spoken languages.To address these challenges, we propose a multilingual gloss-free model withdual CTC objectives for token-level SL identification and spoken textgeneration. Our model supports 10 SLs and handles one-to-one, many-to-one, andmany-to-many SLT tasks, achieving competitive performance compared tostate-of-the-art methods on three widely adopted benchmarks: multilingualSP-10, PHOENIX14T, and CSL-Daily.

 

Quick Read (beta)

loading the full paper ...