Abstract
Recently, video object segmentation (VOS) referred by multi-modal signals,e.g., language and audio, has evoked increasing attention in both industry andacademia. It is challenging for exploring the semantic alignment withinmodalities and the visual correspondence across frames. However, existingmethods adopt separate network architectures for different modalities, andneglect the inter-frame temporal interaction with references. In this paper, wepropose MUTR, a Multi-modal Unified Temporal transformer for Referring videoobject segmentation. With a unified framework for the first time, MUTR adopts aDETR-style transformer and is capable of segmenting video objects designated byeither text or audio reference. Specifically, we introduce two strategies tofully explore the temporal relations between videos and multi-modal signals.Firstly, for low-level temporal aggregation before the transformer, we enablethe multi-modal references to capture multi-scale visual cues from consecutivevideo frames. This effectively endows the text or audio signals with temporalknowledge and boosts the semantic alignment between modalities. Secondly, forhigh-level temporal interaction after the transformer, we conduct inter-framefeature communication for different object embeddings, contributing to betterobject-wise correspondence for tracking along the video. On Ref-YouTube-VOS andAVSBench datasets with respective text and audio references, MUTR achieves+4.2% and +4.2% J&F improvements to state-of-the-art methods, demonstrating oursignificance for unified multi-modal VOS. Code is released athttps://github.com/OpenGVLab/MUTR.