Abstract
The success of building textless speech-to-speech translation (S2ST) modelshas attracted much attention. However, S2ST still faces two main challenges: 1)extracting linguistic features for various speech signals, called cross-modal(CM), and 2) learning alignment of difference languages in long sequences,called cross-lingual (CL). We propose the unit language to overcome the twomodeling challenges. The unit language can be considered a text-likerepresentation format, constructed using $n$-gram language modeling. Weimplement multi-task learning to utilize the unit language in guiding thespeech modeling process. Our initial results reveal a conflict when applyingsource and target unit languages simultaneously. We propose task promptmodeling to mitigate this conflict. We conduct experiments on four languages ofthe Voxpupil dataset. Our method demonstrates significant improvements over astrong baseline and achieves performance comparable to models trained withtext.