Object-aware Video-language Pre-training for Retrieval

  • 2021-12-01 17:06:39
  • Alex Jinpeng Wang, Yixiao Ge, Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, Mike Zheng Shou
  • 0

Abstract

Recently, by introducing large-scale dataset and strong transformer network,video-language pre-training has shown great success especially for retrieval.Yet, existing video-language transformer models do not explicitly fine-grainedsemantic align. In this work, we present Object-aware Transformers, anobject-centric approach that extends video-language transformer to incorporateobject representations. The key idea is to leverage the bounding boxes andobject tags to guide the training process. We evaluate our model on threestandard sub-tasks of video-text matching on four widely used benchmarks. Wealso provide deep analysis and detailed ablation about the proposed method. Weshow clear improvement in performance across all tasks and datasets considered,demonstrating the value of a model that incorporates object representationsinto a video-language architecture. The code will be released at\url{https://github.com/FingerRec/OA-Transformer}.

 

Quick Read (beta)

loading the full paper ...