VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research

  • 2019-04-06 16:50:31
  • Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, William Yang Wang
  • 4

Abstract

We present a new large-scale multilingual video description dataset, VATEX,which contains over 41,250 videos and 825,000 captions in both English andChinese. Among the captions, there are over 206,000 English-Chinese paralleltranslation pairs. Compared to the widely-used MSR-VTT dataset, VATEX ismultilingual, larger, linguistically complex, and more diverse in terms of bothvideo and natural language descriptions. We also introduce two tasks forvideo-and-language research based on VATEX: (1) Multilingual Video Captioning,aimed at describing a video in various languages with a compact unifiedcaptioning model, and (2) Video-guided Machine Translation, to translate asource language description into the target language using the videoinformation as additional spatiotemporal context. Extensive experiments on theVATEX dataset show that, first, the unified multilingual model can not onlyproduce both English and Chinese descriptions for a video more efficiently, butalso offer improved performance over the monolingual models. Furthermore, wedemonstrate that the spatiotemporal video context can be effectively utilizedto align source and target languages and thus assist machine translation. Inthe end, we discuss the potentials of using VATEX for other video-and-languageresearch.

 

Quick Read (beta)

VaTeX: A Large-Scale, High-Quality Multilingual Dataset
for Video-and-Language Research

Xin Wang 1  Jiawei Wufootnotemark:  1  Junkun Chen2  Lei Li2  Yuan-Fang Wang1  William Yang Wang1
1University of California, Santa Barbara, CA, USA
2ByteDance AI Lab, Beijing, China
{xwang,jiawei_wu,yfwang,william}@cs.ucsb.edu  {chenjunkun,lileilab}@bytedance.com
Equal contribution.
Abstract

We present a new large-scale multilingual video description dataset, VaTeX 11 1 VaTeX stands for Video And TEXt, where X also represents various languages., which contains over 41,250 videos and 825,000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSR-VTT dataset [63], VaTeX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on VaTeX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the VaTeX dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using VaTeX for other video-and-language research.

(a) Multilingual Video Captioning
(b) Video-guided Machine Translation
Figure 1: Demonstration of the VaTeX tasks. (a) A compact unified video captioning model is required to accurately describe the video content in both English and Chinese. (b) The machine translation model mistakenly interprets “pull up bar” as “pulling pub” and “do pull ups” as “do pull” (two verbs), which are meaningless. While with the relevant video context, the English sentence is precisely translated into Chinese.

1 Introduction

Recently, researchers in both computer vision and natural language processing communities are striving to bridge videos and natural language. For a deeper understanding of the activities, the task of video captioning/description aims at describing the video content with natural language. A few datasets have been introduced for this task and cover a variety of domains, such as cooking [16, 68], movie [46], human actions [14, 63], and social media [22]. Despite the variants of this task, the fundamental challenge is to accurately depict the important activities in a video clip, which requires high-quality, diverse captions that describe a wide variety of videos at scale. Moreover, existing large-scale video captioning datasets are mostly monolingual (English only) and thus the development of video captioning models is restricted to English corpora. However, the study of multilingual video captioning is essential for a large population on the planet who cannot speak English.

Figure 2: A sample of our VaTeX dataset. The video has 10 English and 10 Chinese descriptions. All depicts the same video and thus are distantly parallel to each other, while the last five are the paired translations to each other.

To this end, we collect a new and largest multilingual dataset for video-and-language research, VaTeX, that contains over 41,250 unique videos and 825,000 high-quality captions. It covers 600 human activities and a variety of video content. Each video is paired with 10 English and 10 Chinese diverse captions from 20 individual human annotators. Figure 2 illustrates a sample of our VaTeX dataset. Compared to the most popular large-scale video description dataset MSR-VTT [63], VaTeX is characterized by the following major unique properties. First, it contains both English and Chinese descriptions at scale, which can support many multilingual studies that are constrained by monolingual datasets. Secondly, VaTeX has the largest number of clip-sentence pairs with each video clip annotated with multiple unique sentences, and every caption is unique in the whole corpus. Thirdly, VaTeX contains more comprehensive yet representative video content, covering 600 human activities in total. Furthermore, both the English and Chinese corpora in VaTeX are lexically richer and thus can empower more natural and diverse caption generation.

With the capabilities of the VaTeX dataset, we introduce the task of multilingual video captioning (see Figure 0(a)), which is to train a unified model to generate video descriptions in multiple languages (e.g., English and Chinese). However, would the multilingual knowledge further reinforce video understanding? We examine different multilingual models where different portions of the architectures are shared for multiple languages. Experiments show that a compact unified multilingual captioning model is not only more efficient but also more effective than monolingual models.

Video captioning is designed to push forward video understanding with natural language descriptions, but can video information help natural language tasks like machine translation in return? To answer this question, we collect around 206K English-Chinese parallel sentences among all the captions and introduce a new task, video-guided machine translation (VMT), to translate a source language description into the target language using the video information as additional spatiotemporal context. We assume that the spatiotemporal context would reduce the ambiguity of languages (especially for verbs and nouns) and hence promote the alignment between language pairs. So we further conduct extensive experiments and verify the effectiveness of VMT. In Figure 0(b), we demonstrate an example where video information can play a crucial role in translating essential information.

In summary, our contributions are mainly three-fold:

  • We collect a new large-scale and high-quality multilingual video description dataset for the advance of the video-and-language research, and conduct in-depth comparisons among MSR-VTT, VaTeX English corpus, and VaTeX Chinese corpus.

  • We introduce the task of multilingual video captioning and validate its efficiency and effectiveness of generating video descriptions in both English and Chinese with a compact, unified model.

  • We are the first to propose the task of video-guided machine translation and examine the effectiveness of incorporating spatiotemporal context to improve the performance of machine translation.

2 Related Work

Video Description Datasets. Various datasets for video description/captioning have been introduced to empower different ways to describe the video content, covering a wide range of domains, such as cooking [16, 68, 44, 45], movie [54, 46, 47], social media [22], and human activities [14, 50, 63, 30]. In Table 1, we summarize existing video description datasets [1] and briefly compare their major statistics. Generally, video description tasks can mainly be divided into two families, single-sentence generation (e.g., [14, 63]) and multi-sentence generation (e.g., [30]), though they may appear as different variants due to the difference of the corpora, e.g., video title generation [66] and video story generation [22]. In this work, we present a large-scale, high-quality multilingual benchmark for single-sentence generation, aiming at encouraging fundamental approaches towards a more in-depth understanding of human actions. As shown in Table 1, our VaTeX dataset is the largest benchmark in terms of video coverage and the language corpora; it also provides 20 captions for each video clip to take into consideration human variance when describing the same video and hence supports more human-consistent evaluations. Moreover, our VaTeX dataset contains both English and Chinese descriptions at scale, which is an order of magnitude larger than MSVD [14]. Besides, MSVD does not have any translation pairs as VaTeX does. Therefore, VaTeX can empower many multilingual, multimodal research that requires the large-scale training.

Dataset MLingual Domain #classes #videos:clips #sent #sent/clip
TACoS[44] - cooking 26 127:3.5k 11.8k -
TACoS-MLevel[45] - cooking 67 185:25k 75k 3
Youcook[16] - cooking 6 88:- 2.7k -
Youcook II[68] - cooking 89 2k:15.4k 15.4k 1
MPII MD[46] - movie - 94:68k 68.3k 1
M-VAD[54] - movie - 92:46k 55.9k -
LSMDC[47] - movie - 200:128k 128k 1
Charades[50] - indoor 157 10k:10k 27.8k 2-3
VideoStory[22] - social media - 20k:123k 123k 1
ActyNet-Cap[30] - open 200 20k:100k 100k 1
MSVD[14] open - 2k:2k 70k 35
VTW[66] - open - 18k:18k 18k 1
MSR-VTT[63] - open 257 7k:10k 200k 20
VaTeX (ours) open 600 41.3k:41.3k 826k 20
Table 1: Comparison of the video description datasets.

Multilingual Visual Understanding. Numerous tasks have been proposed to combine vision and language to enhance the understanding of either or both, such as video/image captioning [18, 58, 2], visual question answering (VQA) [4], and natural language moment retrieval [25], etc. Multilingual studies are rarely explored in the vision and language domain. Gao et al[21] introduce a multilingual image question answering dataset, and Shimizu et al[49] propose a cross-lingual method for making use of English annotations to improve a Japanese VQA system. Pappas et al[41] propose multilingual visual concept clustering to study the commonalities and differences among different languages. Meanwhile, multilingual image captioning is introduced to describe the content of an image with multiple languages [32, 55, 33]. However, none of them study the interaction between videos and multilingual knowledge. Therefore, we collect a multilingual dataset for video-and-language research and introduce the task of multilingual video captioning to facilitate multilingual understanding of the video dynamics.

Multimodal Machine Translation. The multimodal machine translation task aims at generating a better target sentence by supplementing the source sentence with extra information gleaned from other modalities. Previous studies mainly focus on using images as the visual modality to help machine translation [52, 19, 6]. The Multi30K dataset [20], which is annotated based on the image captioning dataset Flickr30K [43], is commonly used in this direction. For instance, [27, 23] consider the object features of the images, and  [10, 34] import convolutional image features into machine translation. Additionally, other studies [38, 12, 39, 9] explore the cross-modal feature fusion of images and sentences. In this work, we are the first to consider videos as the spatiotemporal context for machine translation and introduce a new task—video-guided machine translation. Compared with images, videos provide richer visual information like actions and temporal transitions, which can better assist models in understanding and aligning the words/phrases between the source and target languages. Moreover, the parallel captions in VaTeX go beyond of spatial relations and are more linguistically complex than Multi30K, e.g., a series of actions. Last but not least, our VaTeX dataset contains over 206K English-Chinese sentence pairs (5 per video), which is approximately seven times larger than Multi30K.

3 VaTeX Dataset

Split train validation public test secret test
#videos 25,991 3,000 6,000 6,278
action label - -
Table 2: The splits of the VaTeX dataset (✓ indicates the videos have publicly accessible action labels). For the secret test set, we holdout the human-annotated captions for challenge use.
(a) Distributions of caption lengths.
(b) Distributions of unique nouns per caption.
(c) Distributions of unique verbs per caption.
Figure 3: Statistical histogram distributions on MSR-VTT, VaTeX-en, and VaTeX-zh. Compared to MSR-VTT, the VaTeX dataset contains longer captions, each with more unique nouns and verbs.

3.1 Data Collection

For a wide coverage of human activities, we reuse a subset of the videos from the Kinetics-600 dataset [28], the largest and widely-used benchmark for action classification. Kinetics-600 contains 600 human action classes and around half a million video clips. To collect those videos, Kay et al[28] first built an action list by combining previous video datasets [24, 31, 51, 3, 60], and then searched the videos from YouTube for candidates, which eventually were filtered by Amazon Mechanical Turkers. Each clip lasts around 10 seconds and is taken from a unique YouTube video. The VaTeX dataset connects videos to natural language descriptions rather than coarse action labels. Notably, we collect the English and Chinese descriptions of 41,269 valid video clips from the Kinetics-600 validation and holdout test sets, costing approximately $51,000 in total. The data collection window is around two months. We have obtained approvals from the institutional reviewing agency to conduct human subject crowdsourcing experiments, and our payment rate is reasonably high (the estimated hourly rate is higher than the minimum wage required by law).

We split those videos into four sets as shown in Table 2. Note that the train and validation sets are split from the Kinetics-600 validation set, and the test sets are from the Kinetics-600 holdout test set. Below we detail the collection process of both English and Chinese descriptions.

3.1.1 English Description Collection

Towards large-scale and diverse human-annotated video descriptions, we build upon Amazon Mechanical Turk (AMT)22 2 https://www.mturk.com and collect 10 English captions for every video clip in VaTeX, where each caption from an individual worker. Specifically, the workers are required to watch the video clips and describe the corresponding captions in English. In each assignment, the workers are required to describe 5 videos. We show the instructions that the workers should describe all the important people and actions in the video clips with the word count in each caption no less than 10. The AMT interface can be found in the supplementary material, which contains more details.

To ensure the quality of the collected captions, we employ only workers from the English-speaking countries, including Australia, Canada, Ireland, New Zealand, UK, and US. The workers are also required to complete a minimum of 1K previous tasks on AMT with at least a 95% approval rate. Furthermore, we daily spot-check the captions written by each worker to see if they are relevant to the corresponding videos. Meanwhile, we run scripts to check the captions according to the following rules: (1) whether the captions are shorter than 8 words; (2) whether there are repeated captions; (3) whether the captions contain sensitive words; and (4) whether the captions are not written in English. We reject all the captions that do not achieve the requirements and block the workers consistently providing low-quality annotations. The rejected captions are re-collected until all captions strictly follow the requirements. In preliminary experiments, we find that the workers may struggle to write good captions with only the instructions. Hence, we further provide some accepted good examples and rejected bad examples (both are unrelated to the current video clips) for workers’ reference. We observe that this additional information brings in evident quality improvement on the collected captions. Overall, 2,159 qualified workers annotate 412,690 valid English captions.

3.1.2 Chinese Description Collection

Similar to the English corpus, we collect 10 Chinese descriptions for each video. But to support the video-guided machine translation task, we split these 10 descriptions into two parts, five directly describing the video content and the other five are the paired translations of 5 English descriptions for the same video. All annotations are conducted on the Bytedance Crowdsourcing platform33 3 A public Chinese crowdsourcing platform: https://zc.bytedance.com. All workers are native Chinese speakers and have a good education background to guarantee that the video content can be correctly understood and the corresponding descriptions can be accurately written.

duplicated sent rate #unique n-grams #unique POS tags
Dataset sent length intra-video inter-video 1-gram 2-gram 3-gram 4-gram verb noun adjective adverb
MSR-VTT 9.28 66.0% 16.5% 29,004 274,000 614,449 811,903 8,862 19,703 7,329 1,195
VaTeX-en 15.23 0 0 35,589 538,517 1,660,015 2,773,211 12,796 23,288 10,639 1,924
VaTeX-zh 13.95 0 0 47,065 626,031 1,752,085 2,687,166 20,299 30,797 4,703 3,086
Table 3: We demonstrate the average sentence length, the duplicated sentence rate within a video (intra-video) and within the whole corpus (inter-video), the numbers of unique n-grams and POS tags. Our VaTeX dataset is lexically richer than MSR-VTT in general. Note that the Chinese POS tagging rules follow the Penn Chinese Treebank standard [62], which is different from English due to different morphemes. For instance, VaTeX-zh has more nouns and verbs but fewer adjectives than VaTeX-en, because the semantics of many Chinese adjectives are included in nouns or verbs [67]44 4 For example, the segmented Chinese word {CJK*}UTF8gbsn长发 (“long hair”) is labeled as one noun in Chinese, but an adjective (“long”) and a noun (“hair”) in English..

For the first part that directly describes the video content, we follow the same annotation rules as in the collection process of the English captions, except that each Chinese caption must contain at least 15 Chinese characters.

As for the second part, we aim to collect 5 English-Chinese parallel pairs for each video to enable the VMT task. However, the direct translation by professional translators is costly and time-consuming. Thus, following previous methods [8, 65] on collecting parallel pairs, we choose the post-editing annotation strategy. Particularly, for each video, we randomly sample 5 captions from the annotated 10 English captions and use multiple translation systems to translate them into Chinese reference sentences. Then the annotation task is, given the video and the references, the workers are required to post-edit the references and write the parallel Chinese sentence following two rules: (1) the original sentence structure and semantics need be maintained to guarantee the alignment to the corresponding English sentence, and (2) lost or wrong entities and actions could be corrected based on the video content to eliminate the errors from the translation systems. To further reduce the annotation bias towards one specific translation system, here we use three advanced EnglishChinese translation systems (Google, Microsoft, and self-developed translation systems) to provide the workers with machine-translated sentences as references for each English caption.

In order to ensure the quality of the Chinese captions, we conduct a strict two-stage verification: every collected description must be reviewed and approved by another independent worker. Workers with less than 90% approval rate are blocked. The interfaces for Chinese caption collection can be found in the supplementary material. Eventually, 450 Chinese workers participate in these two tasks and write 412,690 valid Chinese captions. Half of the captions are English-Chinese parallel sentences, so we have 206,345 translation pairs in total.

3.2 Dataset Analysis

In Table 1, we briefly compare the overall statistics of the existing video description datasets. In this section, we conduct comprehensive analysis between our VaTeX dataset and the MSR-VTT dataset [63], which is the widely-used benchmark for video captioning and the closest to VaTeX in terms of domain and scale. Since MSR-VTT only has English corpus, we split VaTeX into the English corpus (VaTeX-en) and the Chinese corpus (VaTeX-zh) for comparison. VaTeX contains 413k English and 413k Chinese captions depicting 41.3k unique videos from 600 activities, while MSR-VTT has 200k captions describing 7k videos from 257 activities. In addition to the larger scale, the captions in both VaTeX-en and VaTeX-zh are longer and more detailed than those in MSR-VTT (see Figure 3). The average caption lengths of VaTeX-en, VaTeX-zh, and MSR-VTT are 15.23, 13.95, and 9.28.

To assess the linguistic complexity, we compare the unique n-grams and part-of-speech (POS) tags (e.g., verb, noun, adverb etc.) among MSR-VTT, VaTeX-en and VaTeX-zh (see Table 4), which illustrates the improvement of VaTeX over MSR-VTT and the difference between the English and Chinese corpora. Evidently, our VaTeX datasets represent a wider variety of caption styles and cover a broader range of actions, objects, and visual scenes.

We also perform in-depth comparisons of caption diversity. First, as seen in Table 4, MSR-VTT faces a severe duplication issue in that 66.0% of the videos contains some exactly same captions, while our VaTeX datasets are free of this problem and guarantee that the captions within the same video are unique. Not only within videos, but the captions in our VaTeX datasets are also much more diverse even within the whole corpus, which indicates that our VaTeX can also be a high-quality benchmark for video retrieval.

Figure 4: Type-caption curves. Type: unique 4-gram. VaTeX has more lexical styles and caption diversity than MSR-VTT.

For a more intuitive measure of the lexical richness and caption diversity, we then propose the Type-Caption Curve, which is adapted from the type-token vocabulary curve [64] but specially designed for the caption corpora here. The total number of captions and the number of distinct vocabulary words (types) are computed for each corpus. So we plot the number of types against the number of captions for MSR-VTT, VaTeX-en, and VaTeX-zh (see Figure 4 where we choose 4-grams as the types). From these type-caption curves, inferences are drawn about lexical style or caption diversity (vocabulary use), as well as lexical competence (vocabulary size), so our VaTeX datasets are shown to be more linguistically complex and diverse.

4 VaTeX Tasks

4.1 Multilingual Video Captioning

Multilingual video captioning is the task of describing the content of a video using more than one language such as English and Chinese. Below we first introduce a baseline model for monolingual video captioning and then present three different models for multilingual video captioning.

Figure 5: Monolingual video captioning model.

4.1.1 Models

We begin with the well-known attention-based encoder-decoder model for video captioning. As illustrated in Figure 5, there are three main modules to this architecture:

  • A 3D convolutional neural network (3D ConvNet) that learns the spatiotemporal features of the video and outputs a sequence of segment-level features X={x1,x2,,xL}.

  • A video encoder module fenc that encodes X into video-level features V={v1,v2,,vL} by modeling long-range temporal contexts.

  • An attention-based language decoder module fdec that produces a word yt at every time step t by considering the word at previous step yt-1, the visual context vector ct learned from the attention mechanism.

We instantiate the captioning model by adapting the model architectures from the state-of-the-art video captioning methods [42, 59]. We employ the pretrained I3D model [13] for action recognition as the 3D ConvNet to obtain the visual features X, Bidirectional LSTM [48] (bi-LSTM) as the video encoder fenc, and LSTM [26] as the language decoder fdec. We also adopt the dot-product attention, so at the decoding step t, we have

yt,ht=fdec([yt-1,ct],ht-1), (1)

where ht is the hidden state of the decoder at step t and

ct =softmax(ht-1WVT)V, (2)

where W is a learnable projection matrix.

To enable multilingual video captioning, we examine three methods (see Figure 6): (1) Two Base models, which are two monolingual encoder-decoder models (as described in Figure 5) trained separately for either English or Chinese; (2) A Shared Enc model, which has a shared video encoder but two language decoders to generate English and Chinese; (3) A Shared Enc-Dec model, where there are just one encoder and one decoder, both shared by English and Chinese, and the only difference is that the word embedding weight matrices are different for different languages.

Figure 6: Multilingual video captioning models.

4.1.2 Experimental Setup

Data Preprocessing. We train the models on the VaTeX dataset following the splits in Table 2. To preprocess the videos, we sample each video at 25fps and extract the I3D features [13] from these sampled frames. The I3D model is pretrained on the original Kinetics training dataset [28] and used here without fine-tuning. Both the English and Chinese captions are truncated to a maximum of 30 words. Note that we use the segmented Chinese words55 5 We use the open source tool Jieba for Chinese word segmentation: https://github.com/fxsjy/jieba rather than raw Chinese characters. The vocabularies are built with a minimum word count 5, resulting in around 11,000 English words and about 14,000 Chinese words.

Implementation Details. All the hyperparameters are tuned on the validation sets and same for both English and Chinese caption training. The video encoder is a bi-LSTM of size 512 and the decoder LSTM is of size 1024. The dimensions of the word embedding layers are 512. All models are trained using MLE loss and optimized using Adam optimizer [29] with a batch size 512. We adopt Dropout for regularization. The learning rate is initially set as 0.001 and then halted when the current CIDEr score does not surpass the previous best for 4 epochs. Schedule sampling [7] is employed to train the models. At test time, we use beam search of size 5 to report the final results.

English Chinese

Model
#Params B-1 B-2 B-3 B-4 Meteor Rouge CIDEr B-1 B-2 B-3 B-4 Meteor Rouge CIDEr
Base w/o WT 52.5M 70.8 52.7 38.9 27.8 21.7 46.7 45.4 74.3 53.6 36.6 25.2 29.8 51.8 35.7
Base 39.7M 71.1 53.2 39.3 28.2 21.7 46.9 45.7 74.2 53.6 36.8 25.2 29.7 51.7 36.3
Shared Enc 34.9M 72.0 53.9 40.0 28.9 21.9 47.4 46.8 75.5 54.9 38.1 26.3 29.9 52.2 36.7
Shared Enc-Dec 26.3M 71.7 53.8 39.9 28.7 21.9 47.2 45.6 75.5 55.2 38.1 25.9 29.9 52.5 36.9
Table 4: Multilingual video captioning. We report the results of the baseline models in terms of BLEU (B), Meteor, and Rouge-L, and CIDEr scores. WT: weight tying, which means the input word embedding layer and the softmax layer share the same weight matrix.

Evaluation Metrics. We adopt four diverse automatic evaluation metrics: BLEU [40], Meteor [17], Rouge-L [35], and CIDEr [56]. We use the standard evaluation code from MS-COCO server [15] to obtain the results.

4.1.3 Results and Analysis

Table 4 shows the results of the three baseline models on both English and Chinese test sets. The performances of the multilingual models (Shared Enc and Shared Enc-Dec) are consistently (though not significantly) improved over the monolingual model (Base). It indicates that multilingual learning indeed helps video understanding by sharing the video encoder, as the Shared Enc performs the best. More importantly, the parameters of the Shared Enc and Shared Enc-Dec are significantly reduced by 4.7M and 13.4M over the Base models. These observations validate that a compact unified model is able to produce captions in multiple languages and visual understanding can benefit from multilingual knowledge learning. We also believe that specialized multilingual model design would further improve the understanding of the videos and lead to better results.

4.2 Video-guided Machine Translation

In this section, we discuss the enabled new task, Video-guided Machine Translation (VMT), to translate a source language sentence into the target language using the video information as additional spatiotemporal context. This task has various potential real-world applications, e.g., translating posts with the video content in social media.

4.2.1 Method

In VMT, the translation system takes a source sentence and the corresponding video as the input, and generates the translated target sentence. To effectively utilize the two modalities, text and video, we design a multimodal sequence-to-sequence model [53, 57] with the attention mechanism [5, 37] for VMT. The overview of our model is shown in Figure 7, which mainly consists of the following three modules.

Source Encoder. For each source sentence represented as a sequence of N word embeddings S={s1,s2,,sN}, the source encoder fencsrc transforms it into the sentence features U={u1,u2,,uN}.

Video Encoder. Similar in Section 4.1, we use a 3D ConvNet to convert each video into a sequence of segment-level features X. Then we employ a video encoder fencvi to transform X into the video features V={v1,v2,,vL}.

Target Decoder. The sentence embedding from the source language encoder fencsrc and the video embedding from the video encoder fencvi are concatenated and fed into the target language decoder fdectgt. To dynamically highlight the important words of the source sentence and the crucial spatiotemporal segments in the video, we equip the target decoder fdecsrc with two attention mechanisms. Thus, at each decoding step t, we have

yt,ht=fdectgt([yt-1,ctsrc,ctvi],ht-1), (3)

where ht is the hidden state of the decoder at step t. ctvi is the video context vector that is computed with the temporal attention of the video segments (see Equation 2), and ctsrc is the source language context vector:

ctsrc=softmax(ht-1WsrcUT)U, (4)

where Wsrc is a learnable projection matrix.

Figure 7: Video-guided machine translation model.

4.2.2 Experimental Setup

Baselines. We consider the following three baselines to compare: (1) Base NMT Model: We only consider the text information for machine translation and adopt the encoder-decoder model with the source attention mechanism. (2) Average Video Features: We average the segment-level features X of each video as x¯. The average video feature x¯ is then concatenated with each word embedding st in S. The model structure is the same as the base NMT model. (3) LSTM Video Features: This is our VMT model without the temporal attention for videos in the decoder.

Implementation Details. The data prepossessing steps are the same as in the Section 4.1.2 except that we truncate the captions with maximum length 40 here. The baseline NMT is composed of a 2-layer bi-LSTM encoder of size 512 and a 2-layer LSTM decoder of size 1024. The dimensions of both English and Chinese word embeddings are 512. The video encoder is a bi-LSTM of size 512. MLE loss is implemented to train the model using Adam optimizer [29]. The batch size is 32 during training and early-stopping is used to choose the models. As for evaluation, we use beam search of size 5 to report the results on the BLEU-4 metric.

Model EnglishChinese ChineseEnglish
Base NMT w/o VI 27.74 (+0.00) 25.36 (+0.00)
+ Average VI 27.82 (+0.08) 25.39 (+0.03)
+ LSTM VI w/o Attn 28.45 (+0.71) 25.61 (+0.25)
+ LSTM VI w/ Attn (VMT) 30.20 (+2.46) 27.18 (+1.82)
Table 5: Video-guided Machine Translation. We report the results of the baseline models in terms of the BLEU-4 score. VI: Video features from the pretrained I3D model. Attn: temporal attention mechanism.

4.2.3 Results and Analysis

VMT. We first show in Table 5 the results of four different models on EnglishChinese and ChineseEnglish translations. The marginal improvements by the Average video Features and the LSTM Video Features reveal that, passively receiving and incorporating the video features is ineffective in helping align source and target languages. However, we can observe that the translation system achieves much better performance when using the LSTM Video Features with temporal attention (our full VMT model) to dynamically interact with the video features. It is because that with the attention mechanism, the language dynamics are used as a query to highlight the relevant spatiotemporal features in the video, and then the learned video context would assist the word mapping between source and target language spaces. This also validates that extra video information can be effectively utilized to boost machine translation systems.

Masked VMT. Videos contain rich information on subject nouns, object nouns, and action verbs. Therefore, we further conduct noun/verb masking experiments [11] to investigate to what extent the video information can help machine translation. We randomly replace 0%/25%/50%/75%/100% nouns or verbs of the English captions with a special token [M], and then train the NMT and VMT models on the EnglishChinese translation task with different masking rates. This experimental design is to evaluate the capability of VMT in recovering the missing information of the source sentence with the help of the video context.

In addition to the BLEU-4 metric, we propose to use the noun/verb recovery accuracy, which is the percentages of the correctly translated nouns/verbs in the target sentences, to precisely evaluate the impact of additional video information on recovering nouns/verbs. The results with different masking rates are shown in Table 6. First, the VMT model consistently outperforms the NMT model with different masking rates on both metrics. Moreover, as the masking rate increases, the NMT model struggles to figure out the correct nouns/verbs because of the scarce parallel caption pairs; while the VMT model can rely on the video context to obtain more useful information for translation, and thus the performance gap on the recovery accuracy increases dramatically. It shows that in our VMT model, video information can play a crucial role in understanding subjects, objects, and actions, as well as their relations.

BLEU-4 Accuracy (%)
Rate 0% 25% 50% 75% 100% 0% 25% 50% 75% 100%
Noun Masking
NMT 27.7 21.5 14.3 9.4 4.7 70.8 55.4 36.1 17.5 11.6
VMT 30.2 25.6 19.8 17.6 15.6 78.6 68.2 52.9 46.4 41.3
Verb Masking
NMT 27.7 23.9 16.4 12.5 8.7 66.4 58.0 42.8 34.7 21.5
VMT 30.2 28.8 23.7 20.8 18.6 71.5 65.3 56.1 50.2 42.6
Table 6: Video-guided machine translation on EnglishChinese with different noun/verb masking rates. We evaluate the results using the BLEU-4 score and noun/verb recovery accuracy.

5 Discussion and Future Work

In this paper, we introduce a new large-scale multilingual dataset for video-and-language research. In addition to (multilingual) video captioning and video-guided machine translation, there are also some other potentials of this dataset. For example, since the natural language descriptions in VaTeX are unique and each one corresponds to a video clip, one promising direction is to retrieve video clips using multilingual descriptions of our dataset as textual queries [36]. Meanwhile, VaTeX has 600 fine-grained action labels, so we can further holdout certain action classes to evaluate the generalization of different video captioning models to support zero-/few-shot learning [61]. Furthermore, our dataset can possibly contribute to other research fields like Neuroscience. For instance, when describing the same videos, the focus points of people using different languages can be reflected by their written captions. By analyzing multilingual captions, one can likely infer the commonality and discrepancy on the brain attention of people with different cultural and linguistic backgrounds. In general, we hope the release of our VaTeX dataset would facilitate the advance of video-and-language research.

Acknowledgement

This work is collaborated with and funded by the ByteDance AI Lab. We would like to thank the crowdsourcing workers who contribute to the data collection.

Supplementary Material

The supplementary material is organized as follows:

  • Data collection interfaces: We show the AMT interface for English caption collection in Figure 8. Since the Chinese captions are divided into two parts, we build two separate interfaces, one of which is to collect the captions that directly describe the video (Figure 9) and the other for collecting the Chinese translations parallel to the English captions (Figure 10).

  • More VaTeX samples: In addition to the example shown in the main paper, Figure 11 demonstrates more samples of our VaTeX dataset.

  • Qualitative results of multilingual video captioning: Figure 12 illustrates some qualitative examples of multilingual video captioning, where we compare both the English and Chinese results generated by the monolingual models (Base), the multilingual model that shares the video encoder for English & Chinese (Shared Enc), and the multilingual model that shares both the video encoder and the language decoder for English & Chinese (Shared Enc-Dec).

  • Qualitative results of video-guided machine translation (VMT): In Figure 13, we showcase the advantages of the VMT model over the base neural machine translation (NMT) model. Moreover, we further conduct the masked machine translation experiments and qualitatively demonstrate the effectiveness of VMT in recovering nouns or verbs in Figure 14.

References

  • [1] N. Aafaq, S. Z. Gilani, W. Liu, and A. Mian. Video description: A survey of methods, datasets and evaluation metrics. CoRR, abs/1806.00186, 2018.
  • [2] H. Agrawal, K. Desai, X. Chen, R. Jain, D. Batra, D. Parikh, S. Lee, and P. Anderson. nocaps: novel object captioning at scale. arXiv preprint arXiv:1812.08658, 2018.
  • [3] M. Andriluka, L. Pishchulin, P. V. Gehler, and B. Schiele. 2d human pose estimation: New benchmark and state of the art analysis. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3686–3693, 2014.
  • [4] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pages 2425–2433, 2015.
  • [5] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference for Learning Representations (ICLR), 2015.
  • [6] L. Barrault, F. Bougares, L. Specia, C. Lala, D. Elliott, and S. Frank. Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation (WMT), pages 304–323, 2018.
  • [7] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 29th Conference on Neural Information Processing Systems (NeurIPS), 2015.
  • [8] H. Bouamor, H. Alshikhabobakr, B. Mohit, and K. Oflazer. A human judgement corpus and a metric for arabic mt evaluation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 207–213, 2014.
  • [9] O. Caglayan, W. Aransa, A. Bardet, M. García-Martínez, F. Bougares, L. Barrault, M. Masana, L. Herranz, and J. van de Weijer. Lium-cvc submissions for wmt17 multimodal translation task. In Proceedings of the 2nd Conference on Machine Translation (WMT), pages 432–439, 2017.
  • [10] O. Caglayan, L. Barrault, and F. Bougares. Multimodal attention for neural machine translation. arXiv preprint arXiv:1609.03976, 2016.
  • [11] O. Caglayan, P. Madhyastha, L. Specia, and L. Barrault. Probing the need for visual context in multimodal machine translation. In Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT, 2019.
  • [12] I. Calixto and Q. Liu. Incorporating global visual features into attention-based neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 992–1003, 2017.
  • [13] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4733, 2017.
  • [14] D. L. Chen and W. B. Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL), 2011.
  • [15] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. CoRR, abs/1504.00325, 2015.
  • [16] P. Das, C. Xu, R. F. Doell, and J. J. Corso. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In Proceedings of the 2013 IEEE conference on computer vision and pattern recognition (CVPR), pages 2634–2641, 2013.
  • [17] M. Denkowski and A. Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the 9th workshop on statistical machine translation, pages 376–380, 2014.
  • [18] J. Donahue, L. A. Hendricks, M. Rohrbach, S. Venugopalan, S. Guadarrama, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:677–691, 2015.
  • [19] D. Elliott, S. Frank, L. Barrault, F. Bougares, and L. Specia. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation (WMT), pages 215–233, September 2017.
  • [20] D. Elliott, S. Frank, K. Sima’an, and L. Specia. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70–74, 2016.
  • [21] H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? dataset and methods for multilingual image question answering. In Proceedings of the 29th Conference on Neural Information Processing Systems (NeurIPS), 2015.
  • [22] S. Gella, M. Lewis, and M. Rohrbach. A dataset for telling the stories of social media videos. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 968–974, 2018.
  • [23] S.-A. Grönroos, B. Huet, M. Kurimo, J. Laaksonen, B. Merialdo, P. Pham, M. Sjöberg, U. Sulubacak, J. Tiedemann, R. Troncy, et al. The memad submission to the wmt18 multimodal translation task. In Proceedings of the 3rd Conference on Machine Translation (WMT), 2018.
  • [24] F. C. Heilbron, V. Escorcia, B. Ghanem, and J. C. Niebles. Activitynet: A large-scale video benchmark for human activity understanding. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 961–970, 2015.
  • [25] L. A. Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. C. Russell. Localizing moments in video with natural language. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), pages 5804–5813, 2017.
  • [26] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9:1735–1780, 1997.
  • [27] P.-Y. Huang, F. Liu, S.-R. Shiang, J. Oh, and C. Dyer. Attention-based multimodal neural machine translation. In Proceedings of the 1st Conference on Machine Translation (WMT), pages 639–645, 2016.
  • [28] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.
  • [29] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations (ICLR), 2015.
  • [30] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Niebles. Dense-captioning events in videos. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), pages 706–715, 2017.
  • [31] H. Kuehne, H. Jhuang, E. Garrote, T. A. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. Proceedings of the 2011 International Conference on Computer Vision (ICCV), pages 2556–2563, 2011.
  • [32] W. Lan, X. Li, and J. Dong. Fluency-guided cross-lingual image captioning. In Proceedings of the 25th ACM international conference on Multimedia (ACM-MM), 2017.
  • [33] X. Li, X. Wang, C. Xu, W. Lan, Q. Wei, G. Yang, and J. Xu. Coco-cn for cross-lingual image tagging, captioning and retrieval. IEEE Transactions on Multimedia, 2019.
  • [34] J. Libovickỳ and J. Helcl. Attention strategies for multi-source sequence-to-sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 196–202, 2017.
  • [35] C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), 2004.
  • [36] D. Lin, S. Fidler, C. Kong, and R. Urtasun. Visual semantic search: Retrieving videos via complex textual queries. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2657–2664, 2014.
  • [37] T. Luong, H. Pham, and C. D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421, 2015.
  • [38] M. Ma, D. Li, K. Zhao, and L. Huang. Osu multimodal machine translation system report. In Proceedings of the 2nd Conference on Machine Translation (WMT), pages 465–469, 2017.
  • [39] P. S. Madhyastha, J. Wang, and L. Specia. Sheffield multimt: Using object posterior predictions for multimodal machine translation. In Proceedings of the 2nd Conference on Machine Translation (WMT), pages 470–476, 2017.
  • [40] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics(ACL), 2001.
  • [41] N. Pappas, M. Redi, M. Topkara, B. Jou, H. Liu, T. Chen, and S.-F. Chang. Multilingual visual sentiment concept matching. In Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval (ICMR), 2016.
  • [42] R. Pasunuru and M. Bansal. Reinforced video captioning with entailment rewards. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.
  • [43] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the 2015 IEEE international conference on computer vision (ICCV), pages 2641–2649, 2015.
  • [44] M. Regneri, M. Rohrbach, D. Wetzel, S. Thater, B. Schiele, and M. Pinkal. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics (TACL), 1:25–36, 2013.
  • [45] A. Rohrbach, M. Rohrbach, W. Qiu, A. Friedrich, M. Pinkal, and B. Schiele. Coherent multi-sentence video description with variable level of detail. In Proceedings of the German conference on pattern recognition, pages 184–195. Springer, 2014.
  • [46] A. Rohrbach, M. Rohrbach, N. Tandon, and B. Schiele. A dataset for movie description. In Proceedings of the 2015 IEEE conference on computer vision and pattern recognition (CVPR), pages 3202–3212, 2015.
  • [47] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele. Movie description. International Journal of Computer Vision (IJCV), 123(1):94–120, May 2017.
  • [48] M. Schuster and K. Paliwal. Bidirectional recurrent neural networks. Trans. Sig. Proc., 45(11):2673–2681, Nov. 1997.
  • [49] N. Shimizu, N. Rong, and T. Miyazaki. Visual question answering dataset for bilingual image understanding: A study of cross-lingual transfer using attention maps. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), 2018.
  • [50] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Proceedings of the 2016 European Conference on Computer Vision (ECCV), pages 510–526, 2016.
  • [51] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402, 2012.
  • [52] L. Specia, S. Frank, K. Sima’an, and D. Elliott. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the 1st Conference on Machine Translation (WMT), pages 543–553, 2016.
  • [53] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Proceedings of the 28th Conference on Neural Information Processing Systems (NeurIPS), pages 3104–3112, 2014.
  • [54] A. Torabi, C. Pal, H. Larochelle, and A. Courville. Using descriptive video services to create a large data source for video annotation research. arXiv preprint arXiv:1503.01070, 2015.
  • [55] S. Tsutsui and D. J. Crandall. Using artificial tokens to control languages for multilingual image caption generation. CoRR, abs/1706.06275, 2017.
  • [56] R. Vedantam, C. L. Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, 2015.
  • [57] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko. Sequence to sequence-video to text. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pages 4534–4542, 2015.
  • [58] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156–3164, 2015.
  • [59] X. Wang, W. Chen, J. Wu, Y.-F. Wang, and W. Y. Wang. Video captioning via hierarchical reinforcement learning. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  • [60] X. Wang, A. Farhadi, and A. Gupta. Actions   transformations. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2658–2667, 2016.
  • [61] X. Wang, J. Wu, D. Zhang, and W. Y. Wang. Learning to compose topic-aware mixture of experts for zero-shot video captioning. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI), 2019.
  • [62] F. Xia. The part-of-speech tagging guidelines for the penn chinese treebank (3.0). IRCS Technical Reports Series, page 38, 2000.
  • [63] J. Xu, T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  • [64] G. Youmans. Measuring lexical style and competence: The type-token vocabulary curve. Style, 24(4):584–599, 1990.
  • [65] W. Zaghouani, N. Habash, O. Obeid, B. Mohit, H. Bouamor, and K. Oflazer. Building an arabic machine translation post-edited corpus: Guidelines and annotation. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC), pages 1869–1876, 2016.
  • [66] K.-H. Zeng, T.-H. Chen, J. C. Niebles, and M. Sun. Generation for user generated videos. In Proceedings of the 2016 European conference on computer vision (ECCV), pages 609–625. Springer, 2016.
  • [67] R. Zhang. Sadness expressions in English and Chinese: Corpus linguistic contrastive semantic analysis. Bloomsbury Publishing, 2016.
  • [68] L. Zhou, C. Xu, and J. J. Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI), 2018.
Figure 8: The AMT interface for collecting the English captions. In each assignment, the workers are required to annotate 5 video clips. The instructions are kept visible for each clip. We provide the workers with the accepted good examples and rejected bad examples to further improve the quality of annotations. Note that the given examples are unrelated to the current video clips.
Figure 9: The interface for collecting the Chinese captions by directly describing the video content. In each assignment, the workers are required to annotate 1 video clip. The instructions are kept visible for each clip. After the first-stage annotation, each Chinese caption must be reviewed and approved by another independent worker.
Figure 10: The interface for collecting the Chinese captions by post-editing the translated reference sentences and watching the video clips. In each assignment, the workers are required to annotate 1 video clip. The instructions are kept visible for each clip. We provide the workers with three reference sentences translated by Google, Microsoft and Self-developed translation systems. Note that the order of three reference sentences is randomly shuffled for each video clip to reduce the annotation bias towards one specific translation system. After the first-stage annotation, each Chinese caption must be reviewed and approved by another independent worker.
Figure 11: More samples of our VaTeX dataset. Each video has 10 English and 10 Chinese descriptions. All depicts the same video and thus are distantly parallel to each other, while the last five are the paired translations to each other.
Figure 12: Qualitative comparison among different methods of multilingual video captioning on the VaTeX dataset. Both the English and Chinese results are shown. For each video sample, we list a human-annotated caption and the generated results by three models, Base, Shared Enc, and Shared Enc-Dec. The multilingual models (Shared Enc and Shared Enc-Dec) can generate more coherent and informative captions than the monolingual model (Base).
Figure 13: Qualitative comparison between neural machine translation (NMT) and video-guided machine translation (VMT) on the VaTeX dataset. For each video sample, we list the original English description and the translated sentences by the base NMT model and our VMT model. The NMT model mistakenly interprets some words and phrases, while the VMT model can generate more precise translation with the corresponding video context.
Figure 14: Qualitative comparison between masked neural machine translation (NMT) and masked video-guided machine translation (VMT) on the VaTeX dataset. The nouns/verbs in English captions are randomly replaced by a special token [M]. For each video sample, we list the original English description and the translated sentences by the base NMT model and our VMT model. The NMT model struggles to figure out the correct nouns/verbs because of the scarce parallel pairs, while the VMT model can rely on the video context to recover the masked nouns/verbs.