Abstract
Large Language Models (LLMs) excel in text-based natural language processingtasks but remain constrained by their reliance on textual inputs and outputs.To enable more natural human-LLM interaction, recent progress have focused onderiving a spoken language model (SLM) that can not only listen but alsogenerate speech. To achieve this, a promising direction is to conductspeech-text joint modeling. However, recent SLM still lag behind text LLM dueto the modality mismatch. One significant mismatch can be the sequence lengthsbetween speech and text tokens. To address this, we introduce Text-AlignedSpeech Tokenization and Embedding (TASTE), a method that directly addresses themodality gap by aligning speech token with the corresponding text transcriptionduring the tokenization stage. We propose a method that can achieve thisthrough the special aggregation mechanism and with speech reconstruction as thetraining objective. We conduct extensive experiments and show that TASTE canpreserve essential paralinguistic information while dramatically reducing thetoken sequence length. Furthermore, by leveraging TASTE, we can adapttext-based LLMs into effective SLMs with parameter-efficient fine-tuningtechniques such as Low-Rank Adaptation (LoRA). Experimental results onbenchmark tasks, including SALMON and StoryCloze, demonstrate that TASTE-basedSLMs perform similarly to previous full-finetuning methods. To our knowledge,TASTE is the first end-to-end approach that utilizes a reconstruction objectiveto automatically learn a text-aligned speech tokenization and embeddingsuitable for spoken language modeling. Our demo, code, and models are publiclyavailable at https://github.com/mtkresearch/TASTE-SpokenLM.