Abstract
Recent advancements in speech-to-speech dialogue systems leverage LLMs formultimodal interactions, yet they remain hindered by fine-tuning requirements,high computational overhead, and text-speech misalignment. Existingspeech-enabled LLMs often degrade conversational quality by modifying the LLM,thereby compromising its linguistic capabilities. In contrast, we proposeLLMVoX, a lightweight 30M-parameter, LLM-agnostic, autoregressive streaming TTSsystem that generates high-quality speech with low latency, while fullypreserving the capabilities of the base LLM. Our approach achieves asignificantly lower Word Error Rate compared to speech-enabled LLMs, whileoperating at comparable latency and UTMOS score. By decoupling speech synthesisfrom LLM processing via a multi-queue token streaming system, LLMVoX supportsseamless, infinite-length dialogues. Its plug-and-play design also facilitatesextension to various tasks with different backbones. Furthermore, LLMVoXgeneralizes to new languages with only dataset adaptation, attaining a lowCharacter Error Rate on an Arabic speech task. Additionally, we have integratedLLMVoX with a Vision-Language Model to create an omni-model with speech, text,and vision capabilities, without requiring additional multimodal training. Ourcode base and project page is available at https://mbzuai-oryx.github.io/LLMVoX .