Abstract
Interacting with the world is a multi-sensory experience: achieving effectivegeneral-purpose interaction requires making use of all available modalities --including vision, touch, and audio -- to fill in gaps from partial observation.For example, when vision is occluded reaching into a bag, a robot should relyon its senses of touch and sound. However, state-of-the-art generalist robotpolicies are typically trained on large datasets to predict robot actionssolely from visual and proprioceptive observations. In this work, we proposeFuSe, a novel approach that enables finetuning visuomotor generalist policieson heterogeneous sensor modalities for which large datasets are not readilyavailable by leveraging natural language as a common cross-modal grounding. Wecombine a multimodal contrastive loss with a sensory-grounded languagegeneration loss to encode high-level semantics. In the context of robotmanipulation, we show that FuSe enables performing challenging tasks thatrequire reasoning jointly over modalities such as vision, touch, and sound in azero-shot setting, such as multimodal prompting, compositional cross-modalprompting, and descriptions of objects it interacts with. We show that the samerecipe is applicable to widely different generalist policies, including bothdiffusion-based generalist policies and large vision-language-action (VLA)models. Extensive experiments in the real world show that FuSeis able toincrease success rates by over 20% compared to all considered baselines.