Abstract
In this paper, we propose a Vision-Audio-Language Omni-peRception pretrainingmodel (VALOR) for multi-modal understanding and generation. Different fromwidely-studied vision-language pretraining models, VALOR jointly modelsrelationships of vision, audio and language in an end-to-end manner. Itcontains three separate encoders for single modality representations, and adecoder for multimodal conditional text generation. We design two pretext tasksto pretrain VALOR model, including Multimodal Grouping Alignment (MGA) andMultimodal Grouping Captioning (MGC). MGA projects vision, language and audioto the same common space, building vision-language, audio-language andaudiovisual-language alignment simultaneously. MGC learns how to generate texttokens in conditions of vision, audio or their both. To promotevision-audio-language pretraining research, we construct a large-scalehigh-quality tri-modality dataset named VALOR-1M, which contains 1M audiablevideos with human annotated audiovisual captions. Extensive experiments showthat VALOR can learn strong multimodal correlations and be generalized tovarious downstream tasks (e.g., retrieval, captioning and question answering),with different input modalities (e.g., vision-language, audio-language andaudiovisual-language). VALOR achieves new state-of-the-art performances onseries of public cross-modality benchmarks. Code and data are available atproject page https://casia-iva-group.github.io/projects/VALOR.