We present a new large-scale multilingual video description dataset, VATEX,which contains over 41,250 videos and 825,000 captions in both English andChinese. Among the captions, there are over 206,000 English-Chinese paralleltranslation pairs. Compared to the widely-used MSR-VTT dataset, VATEX ismultilingual, larger, linguistically complex, and more diverse in terms of bothvideo and natural language descriptions. We also introduce two tasks forvideo-and-language research based on VATEX: (1) Multilingual Video Captioning,aimed at describing a video in various languages with a compact unifiedcaptioning model, and (2) Video-guided Machine Translation, to translate asource language description into the target language using the videoinformation as additional spatiotemporal context. Extensive experiments on theVATEX dataset show that, first, the unified multilingual model can not onlyproduce both English and Chinese descriptions for a video more efficiently, butalso offer improved performance over the monolingual models. Furthermore, wedemonstrate that the spatiotemporal video context can be effectively utilizedto align source and target languages and thus assist machine translation. Inthe end, we discuss the potentials of using VATEX for other video-and-languageresearch.
Quick Read (beta)
VaTeX: A Large-Scale, High-Quality Multilingual Dataset
for Video-and-Language Research
We present a new large-scale multilingual video description dataset, VaTeX 11 1 VaTeX stands for Video And TEXt, where X also represents various languages., which contains over videos and captions in both English and Chinese. Among the captions, there are over English-Chinese parallel translation pairs. Compared to the widely-used MSR-VTT dataset , VaTeX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on VaTeX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the VaTeX dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using VaTeX for other video-and-language research.
Recently, researchers in both computer vision and natural language processing communities are striving to bridge videos and natural language. For a deeper understanding of the activities, the task of video captioning/description aims at describing the video content with natural language. A few datasets have been introduced for this task and cover a variety of domains, such as cooking [16, 72], movie , human actions [14, 66], and social media . Despite the variants of this task, the fundamental challenge is to accurately depict the important activities in a video clip, which requires high-quality, diverse captions that describe a wide variety of videos at scale. Moreover, existing large-scale video captioning datasets are mostly monolingual (English only) and thus the development of video captioning models is restricted to English corpora. However, the study of multilingual video captioning is essential for a large population on the planet who cannot speak English.
To this end, we collect a new large-scale multilingual dataset for video-and-language research, VaTeX, that contains over unique videos and high-quality captions. It covers human activities and a variety of video content. Each video is paired with English and Chinese diverse captions from individual human annotators. Figure 2 illustrates a sample of our VaTeX dataset. Compared to the most popular large-scale video description dataset MSR-VTT , VaTeX is characterized by the following major unique properties. First, it contains both English and Chinese descriptions at scale, which can support many multilingual studies that are constrained by monolingual datasets. Secondly, VaTeX has the largest number of clip-sentence pairs with each video clip annotated with multiple unique sentences, and every caption is unique in the whole corpus. Thirdly, VaTeX contains more comprehensive yet representative video content, covering human activities in total. Furthermore, both the English and Chinese corpora in VaTeX are lexically richer and thus can empower more natural and diverse caption generation.
With the capabilities of the VaTeX dataset, we introduce the task of multilingual video captioning (see Figure 0(a)), which is to train a unified model to generate video descriptions in multiple languages (e.g., English and Chinese). However, would the multilingual knowledge further reinforce video understanding? We examine different multilingual models where different portions of the architectures are shared for multiple languages. Experiments show that a compact unified multilingual captioning model is not only more efficient but also more effective than monolingual models.
Video captioning is designed to push forward video understanding with natural language descriptions, but can video information help natural language tasks like machine translation in return? To answer this question, we collect around 206K English-Chinese parallel sentences among all the captions and introduce a new task, video-guided machine translation (VMT), to translate a source language description into the target language using the video information as additional spatiotemporal context. We assume that the spatiotemporal context would reduce the ambiguity of languages (especially for verbs and nouns) and hence promote the alignment between language pairs. So we further conduct extensive experiments and verify the effectiveness of VMT. In Figure 0(b), we demonstrate an example where video information can play a crucial role in translating essential information.
In summary, our contributions are mainly three-fold:
We collect a new large-scale and high-quality multilingual video description dataset for the advance of the video-and-language research, and conduct in-depth comparisons among MSR-VTT, VaTeX English corpus, and VaTeX Chinese corpus.
We introduce the task of multilingual video captioning and validate its efficiency and effectiveness of generating video descriptions in both English and Chinese with a compact, unified model.
We are the first to propose the task of video-guided machine translation and examine the effectiveness of incorporating spatiotemporal context to improve the performance of machine translation.
2 Related Work
Video Description Datasets. Various datasets for video description/captioning have been introduced to empower different ways to describe the video content, covering a wide range of domains, such as cooking [16, 72, 45, 46], movie [56, 47, 48], social media , and human activities [14, 52, 66, 30]. In Table 1, we summarize existing video description datasets  and briefly compare their major statistics. Generally, video description tasks can mainly be divided into two families, single-sentence generation (e.g., [14, 66]) and multi-sentence generation (e.g., ), though they may appear as different variants due to the difference of the corpora, e.g., video title generation  and video story generation . In this work, we present a large-scale, high-quality multilingual benchmark for single-sentence generation, aiming at encouraging fundamental approaches towards a more in-depth understanding of human actions. As shown in Table 1, our VaTeX dataset is the largest benchmark in terms of video coverage and the language corpora; it also provides captions for each video clip to take into consideration human variance when describing the same video and hence supports more human-consistent evaluations. Moreover, our VaTeX dataset contains both English and Chinese descriptions at scale, which is an order of magnitude larger than MSVD . Besides, MSVD does not have any translation pairs as VaTeX does. Therefore, VaTeX can empower many multilingual, multimodal research that requires large-scale training.
Multilingual Visual Understanding. Numerous tasks have been proposed to combine vision and language to enhance the understanding of either or both, such as video/image captioning [18, 60, 2], visual question answering (VQA) , and natural language moment retrieval , etc. Multilingual studies are rarely explored in the vision and language domain. Gao et al.  introduce a multilingual image question answering dataset, and Shimizu et al.  propose a cross-lingual method for making use of English annotations to improve a Japanese VQA system. Pappas et al.  propose multilingual visual concept clustering to study the commonalities and differences among different languages. Meanwhile, multilingual image captioning is introduced to describe the content of an image with multiple languages [32, 57, 33]. But none of them study the interaction between videos and multilingual knowledge. Sanabria et al.  collect EnglishPortuguese subtitles for the automatic speech recognition (ASR) task, which however do not directly describe the video content. Therefore, we introduce the VaTeX dataset and the task of multilingual video captioning to facilitate multilingual understanding of video dynamics.
Multimodal Machine Translation. The multimodal machine translation task aims at generating a better target sentence by supplementing the source sentence with extra information gleaned from other modalities. Previous studies mainly focus on using images as the visual modality to help machine translation [54, 19, 6]. The Multi30K dataset , which is annotated based on the image captioning dataset Flickr30K , is commonly used in this direction. For instance, [27, 23] consider the object features of the images, and [10, 35] import convolutional image features into machine translation. Additionally, other studies [39, 12, 40, 9] explore the cross-modal feature fusion of images and sentences. In this work, we are the first to consider videos as the spatiotemporal context for machine translation and introduce a new task—video-guided machine translation. Compared with images, videos provide richer visual information like actions and temporal transitions, which can better assist models in understanding and aligning the words/phrases between the source and target languages. Moreover, the parallel captions in VaTeX go beyond of spatial relations and are more linguistically complex than Multi30K, e.g., a series of actions. Last but not least, our VaTeX dataset contains over K English-Chinese sentence pairs ( per video), which is approximately seven times larger than Multi30K.
3 VaTeX Dataset
|Split||train||validation||public test||secret test|
3.1 Data Collection
For a wide coverage of human activities, we reuse a subset of the videos from the Kinetics- dataset , the largest and widely-used benchmark for action classification. Kinetics- contains human action classes and around half a million video clips. To collect those videos, Kay et al.  first built an action list by combining previous video datasets [24, 31, 53, 3, 61], and then searched the videos from YouTube for candidates, which eventually were filtered by Amazon Mechanical Turkers. Each clip lasts around seconds and is taken from a unique YouTube video. The VaTeX dataset connects videos to natural language descriptions rather than coarse action labels. Notably, we collect the English and Chinese descriptions of valid video clips from the Kinetics- validation and holdout test sets, costing approximately $ in total. The data collection window is around two months. We have obtained approvals from the institutional reviewing agency to conduct human subject crowdsourcing experiments, and our payment rate is reasonably high (the estimated hourly rate is higher than the minimum wage required by law).
We split those videos into four sets as shown in Table 2. Note that the train and validation sets are split from the Kinetics- validation set, and the test sets are from the Kinetics- holdout test set. Below we detail the collection process of both English and Chinese descriptions.
3.1.1 English Description Collection
Towards large-scale and diverse human-annotated video descriptions, we build upon Amazon Mechanical Turk (AMT)22 2 https://www.mturk.com and collect English captions for every video clip in VaTeX, where each caption from an individual worker. Specifically, the workers are required to watch the video clips and describe the corresponding captions in English. In each assignment, the workers are required to describe videos. We show the instructions that the workers should describe all the important people and actions in the video clips with the word count in each caption no less than . The AMT interface can be found in the supplementary material, which contains more details.
To ensure the quality of the collected captions, we employ only workers from the English-speaking countries, including Australia, Canada, Ireland, New Zealand, UK, and USA. The workers are also required to complete a minimum of K previous tasks on AMT with at least a approval rate. Furthermore, we daily spot-check the captions written by each worker to see if they are relevant to the corresponding videos. Meanwhile, we run scripts to check the captions according to the following rules: (1) whether the captions are shorter than words; (2) whether there are repeated captions; (3) whether the captions contain sensitive words; and (4) whether the captions are not written in English. We reject all the captions that do not achieve the requirements and block the workers consistently providing low-quality annotations. The rejected captions are re-collected until all captions strictly follow the requirements. In preliminary experiments, we find that the workers may struggle to write good captions with only the instructions. Hence, we further provide some accepted good examples and rejected bad examples (both are unrelated to the current video clips) for workers’ reference. We observe that this additional information brings in evident quality improvement on the collected captions. Overall, qualified workers annotate valid English captions.
3.1.2 Chinese Description Collection
Similar to the English corpus, we collect 10 Chinese descriptions for each video. But to support the video-guided machine translation task, we split these descriptions into two parts, five directly describing the video content and the other five are the paired translations of English descriptions for the same video. All annotations are conducted on the Bytedance Crowdsourcing platform33 3 A public Chinese crowdsourcing platform: https://zc.bytedance.com. All workers are native Chinese speakers and have a good education background to guarantee that the video content can be correctly understood and the corresponding descriptions can be accurately written.
|duplicated sent rate||#unique -grams||#unique POS tags|
For the first part that directly describes the video content, we follow the same annotation rules as in the collection process of the English captions, except that each Chinese caption must contain at least Chinese characters.
As for the second part, we aim to collect English-Chinese parallel pairs for each video to enable the VMT task. However, direct translation by professional translators is costly and time-consuming. Thus, following previous methods [8, 68] on collecting parallel pairs, we choose the post-editing annotation strategy. Particularly, for each video, we randomly sample captions from the annotated English captions and use multiple translation systems to translate them into Chinese reference sentences. Then the annotation task is, given the video and the references, the workers are required to post-edit the references and write the parallel Chinese sentence following two rules: (1) the original sentence structure and semantics need be maintained to guarantee the alignment to the corresponding English sentence, and (2) lost or wrong entities and actions could be corrected based on the video content to eliminate the errors from the translation systems. To further reduce the annotation bias towards one specific translation system, here we use three advanced EnglishChinese translation systems (Google, Microsoft, and self-developed translation systems) to provide the workers with machine-translated sentences as references for each English caption.
In order to ensure the quality of the Chinese captions, we conduct a strict two-stage verification: every collected description must be reviewed and approved by another independent worker. Workers with less than approval rate are blocked. The interfaces for Chinese caption collection can be found in the supplementary material. Eventually, Chinese workers participate in these two tasks and write valid Chinese captions. Half of the captions are English-Chinese parallel sentences, so we have translation pairs in total.
3.2 Dataset Analysis
In Table 1, we briefly compare the overall statistics of the existing video description datasets. In this section, we conduct comprehensive analysis between our VaTeX dataset and the MSR-VTT dataset , which is the widely-used benchmark for video captioning and the closest to VaTeX in terms of domain and scale. Since MSR-VTT only has English corpus, we split VaTeX into the English corpus (VaTeX-en) and the Chinese corpus (VaTeX-zh) for comparison. VaTeX contains k English and k Chinese captions depicting k unique videos from activities, while MSR-VTT has k captions describing k videos from activities. In addition to the larger scale, the captions in both VaTeX-en and VaTeX-zh are longer and more detailed than those in MSR-VTT (see Figure 3). The average caption lengths of VaTeX-en, VaTeX-zh, and MSR-VTT are , , and .
To assess the linguistic complexity, we compare the unique -grams and part-of-speech (POS) tags (e.g., verb, noun, adverb etc.) among MSR-VTT, VaTeX-en and VaTeX-zh (see Table 4), which illustrates the improvement of VaTeX over MSR-VTT and the difference between the English and Chinese corpora. Evidently, our VaTeX datasets represent a wider variety of caption styles and cover a broader range of actions, objects, and visual scenes.
We also perform in-depth comparisons of caption diversity. First, as seen in Table 4, MSR-VTT faces a severe duplication issue in that of the videos contains some exactly same captions, while our VaTeX datasets are free of this problem and guarantee that the captions within the same video are unique. Not only within videos, but the captions in our VaTeX datasets are also much more diverse even within the whole corpus, which indicates that our VaTeX can also be a high-quality benchmark for video retrieval.
For a more intuitive measure of the lexical richness and caption diversity, we then propose the Type-Caption Curve, which is adapted from the type-token vocabulary curve  but specially designed for the caption corpora here. The total number of captions and the number of distinct vocabulary words (types) are computed for each corpus. So we plot the number of types against the number of captions for MSR-VTT, VaTeX-en, and VaTeX-zh (see Figure 4 where we choose -grams as the types). From these type-caption curves, inferences are drawn about lexical style or caption diversity (vocabulary use), as well as lexical competence (vocabulary size), so our VaTeX datasets are shown to be more linguistically complex and diverse.
4 VaTeX Tasks
4.1 Multilingual Video Captioning
Multilingual video captioning is the task of describing the content of a video using more than one language such as English and Chinese. Below we first introduce a baseline model for monolingual video captioning and then present three different models for multilingual video captioning.
We begin with the well-known attention-based encoder-decoder model for video captioning. As illustrated in Figure 5, there are three main modules to this architecture:
A 3D convolutional neural network (3D ConvNet) that learns the spatiotemporal features of the video and outputs a sequence of segment-level features .
A video encoder module that encodes into video-level features by modeling long-range temporal contexts.
An attention-based language decoder module that produces a word at every time step by considering the word at previous step , the visual context vector learned from the attention mechanism.
We instantiate the captioning model by adapting the model architectures from the state-of-the-art video captioning methods [43, 62]. We employ the pretrained I3D model  for action recognition as the 3D ConvNet to obtain the visual features , Bidirectional LSTM  (bi-LSTM) as the video encoder , and LSTM  as the language decoder . We also adopt the dot-product attention, so at the decoding step , we have
where is the hidden state of the decoder at step and
where is a learnable projection matrix.
To enable multilingual video captioning, we examine three methods (see Figure 6): (1) Two Base models, which are two monolingual encoder-decoder models (as described in Figure 5) trained separately for either English or Chinese; (2) A Shared Enc model, which has a shared video encoder but two language decoders to generate English and Chinese; (3) A Shared Enc-Dec model, where there are just one encoder and one decoder, both shared by English and Chinese, and the only difference is that the word embedding weight matrices are different for different languages.
|Base w/o WT||52.5M||28.1 0.38||21.7 0.15||46.8 0.18||44.3 0.98||24.4 0.86||29.6 0.30||51.3 0.43||34.0 0.11|
|Base||39.7M||28.1 0.32||21.6 0.19||46.9 0.16||44.3 0.10||24.9 0.20||29.7 0.21||51.5 0.28||34.7 0.47|
|Shared Enc||34.9M||28.4 0.21||21.7 0.65||47.0 0.09||45.1 0.25||24.9 0.26||29.7 0.11||51.6 0.20||34.9 0.40|
|Shared Enc-Dec||26.3M||27.9 0.50||21.6 0.55||46.8 0.19||44.2 0.23||24.9 0.25||29.8 0.23||51.7 0.09||35.0 0.18|
4.1.2 Experimental Setup
Implementation Details. We train the models on the VaTeX dataset following the splits in Table 2. To preprocess the videos, we sample each video at and extract the I3D features  from these sampled frames. The I3D model is pretrained on the original Kinetics training dataset  and used here without fine-tuning. More details about data preprocessing and implementation can be found in the supplementary material.
4.1.3 Results and Analysis
Table 4 shows the results of the three baseline models on both English and Chinese test sets. The performances of the multilingual models (Shared Enc and Shared Enc-Dec) are consistently (though not significantly) improved over the monolingual model (Base). It indicates that multilingual learning indeed helps video understanding by sharing the video encoder. More importantly, the parameters of the Shared Enc and Shared Enc-Dec are significantly reduced by M and M over the Base models. These observations validate that a compact unified model is able to produce captions in multiple languages and benefits from multilingual knowledge learning. We believe that more specialized multilingual models would improve the understanding of the videos and lead to better results. Furthermore, incorporating multimodal features like audio  would further improve the performance, which we leave for future study.
4.2 Video-guided Machine Translation
In this section, we discuss the enabled new task, Video-guided Machine Translation (VMT), to translate a source language sentence into the target language using the video information as additional spatiotemporal context. This task has various potential real-world applications, e.g., translating posts with the video content in social media.
In VMT, the translation system takes a source sentence and the corresponding video as the input, and generates the translated target sentence. To effectively utilize the two modalities, text and video, we design a multimodal sequence-to-sequence model [55, 59] with the attention mechanism [5, 38] for VMT. The overview of our model is shown in Figure 7, which mainly consists of the following three modules.
Source Encoder. For each source sentence represented as a sequence of word embeddings , the source encoder transforms it into the sentence features .
Video Encoder. Similar in Section 4.1, we use a 3D ConvNet to convert each video into a sequence of segment-level features . Then we employ a video encoder to transform into the video features .
Target Decoder. The sentence embedding from the source language encoder and the video embedding from the video encoder are concatenated and fed into the target language decoder . To dynamically highlight the important words of the source sentence and the crucial spatiotemporal segments in the video, we equip the target decoder with two attention mechanisms. Thus, at each decoding step , we have
where is the hidden state of the decoder at step . is the video context vector that is computed with the temporal attention of the video segments (see Equation 2), and is the source language context vector:
where is a learnable projection matrix.
4.2.2 Experimental Setup
Baselines. We consider the following three baselines to compare: (1) Base NMT Model: We only consider the text information for machine translation and adopt the encoder-decoder model with the source attention mechanism. (2) Average Video Features: We average the segment-level features of each video as . The average video feature is then concatenated with each word embedding in . The model structure is the same as the base NMT model. (3) LSTM Video Features: This is our VMT model without the temporal attention for videos in the decoder.
|Base NMT w/o VI||26.85||24.31|
|LSTM VI w/o Attn||27.43||24.76|
|LSTM VI w/ Attn (VMT)||29.12||26.42|
4.2.3 Results and Analysis
VMT. We first show in Table 5 the results of four different models on ChineseEnglish and EnglishChinese translations. The marginal improvements by the Average video Features and the LSTM Video Features reveal that, passively receiving and incorporating the video features is ineffective in helping align source and target languages. However, we can observe that the translation system achieves much better performance when using the LSTM Video Features with temporal attention (our full VMT model) to dynamically interact with the video features. It is because that with the attention mechanism, the language dynamics are used as a query to highlight the relevant spatiotemporal features in the video, and then the learned video context would assist the word mapping between source and target language spaces. This also validates that extra video information can be effectively utilized to boost machine translation systems.
Masked VMT. Videos contain rich information on subject/object nouns and action verbs. Therefore, we conduct noun/verb masking experiments  to investigate to what extent the video information can help machine translation. We randomly replace nouns or verbs of the English captions with a special token [M], and then train the NMT and VMT models on the ChineseEnglish translation task with different masking rates. This experimental design is to evaluate the capability of VMT in recovering the missing information of the source sentence with the help of the video context.
In addition to the BLEU-4 metric, we propose to use the noun/verb recovery accuracy, which is the percentages of the correctly translated nouns/verbs in the target sentences, to precisely evaluate the impact of additional video information on recovering nouns/verbs. The results with different masking rates are shown in Table 6. First, the VMT model consistently outperforms the NMT model with different masking rates on both metrics. Moreover, as the masking rate increases, the NMT model struggles to figure out the correct nouns/verbs because of the scarce parallel caption pairs; while the VMT model can rely on the video context to obtain more useful information for translation, and thus the performance gap on the recovery accuracy increases dramatically. It shows that in our VMT model, video information can play a crucial role in understanding subjects, objects, and actions, as well as their relations.
5 Discussion and Future Work
In this paper, we introduce a new large-scale multilingual dataset for video-and-language research. In addition to (multilingual) video captioning and video-guided machine translation, there are also some other potentials of this dataset. For example, since the natural language descriptions in VaTeX are unique, one promising direction is to use multilingual descriptions of our dataset as queries to retrieve the video clip from all videos  or even localize it within an untrimmed long video . Meanwhile, VaTeX has 600 fine-grained action labels, so we can holdout certain action classes to evaluate the generalizability of different video captioning models to support zero-/few-shot learning . Furthermore, our dataset can contribute to other research fields like Neuroscience. For instance, when describing the same videos, the focus points of people using different languages can be reflected by their written captions. By analyzing multilingual captions, one can likely infer the commonality and discrepancy on the brain attention of people with different cultural and linguistic backgrounds. In general, we hope the release of our VaTeX dataset would facilitate the advance of video-and-language research.
-  (2018) Video description: a survey of methods, datasets and evaluation metrics. arXiv preprint arXiv:1806.00186. Cited by: §2.
-  (2018) nocaps: novel object captioning at scale. arXiv preprint arXiv:1812.08658. Cited by: §2.
-  (2014) 2D human pose estimation: new benchmark and state of the art analysis. Proceedings of the 27th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3686–3693. Cited by: §3.1.
-  (2015) VQA: visual question answering. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2425–2433. Cited by: §2.
-  (2015) Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference for Learning Representations (ICLR), Cited by: §4.2.1.
-  (2018) Findings of the third shared task on multimodal machine translation. In Proceedings of the 3rd Conference on Machine Translation (WMT), pp. 304–323. Cited by: §2.
-  (2015) Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 29th Conference on Neural Information Processing Systems (NeurIPS), Cited by: Appendix A.
-  (2014) A human judgement corpus and a metric for arabic mt evaluation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 207–213. Cited by: §3.1.2.
-  (2017) LIUM-cvc submissions for wmt17 multimodal translation task. In Proceedings of the 2nd Conference on Machine Translation (WMT), pp. 432–439. Cited by: §2.
-  (2016) Multimodal attention for neural machine translation. arXiv preprint arXiv:1609.03976. Cited by: §2.
-  (2019) Probing the need for visual context in multimodal machine translation. In Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Cited by: §4.2.3.
-  (2017) Incorporating global visual features into attention-based neural machine translation.. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 992–1003. Cited by: §2.
-  (2017) Quo vadis, action recognition? a new model and the kinetics dataset. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724–4733. Cited by: Appendix A, §4.1.1, §4.1.2.
-  (2011) Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §1, Table 1, §2.
-  (2015) Microsoft coco captions: data collection and evaluation server. arXiv preprint arXiv:1504.00325. Cited by: §4.1.2.
-  (2013) A thousand frames in just a few words: lingual description of videos through latent topics and sparse object stitching. In Proceedings of the 26th IEEE conference on computer vision and pattern recognition (CVPR), pp. 2634–2641. Cited by: §1, Table 1, §2.
-  (2014) Meteor universal: language specific translation evaluation for any target language. In Proceedings of the 9th workshop on statistical machine translation, pp. 376–380. Cited by: §4.1.2.
-  (2015) Long-term recurrent convolutional networks for visual recognition and description. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, pp. 677–691. Cited by: §2.
-  (2017) Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the 2nd Conference on Machine Translation (WMT), pp. 215–233. Cited by: §2.
-  (2016) Multi30K: multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pp. 70–74. Cited by: §2.
-  (2015) Are you talking to a machine? dataset and methods for multilingual image question answering. In Proceedings of the 29th Conference on Neural Information Processing Systems (NeurIPS), Cited by: §2.
-  (2018) A dataset for telling the stories of social media videos. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 968–974. Cited by: §1, Table 1, §2.
-  (2018) The memad submission to the wmt18 multimodal translation task. In Proceedings of the 3rd Conference on Machine Translation (WMT), Cited by: §2.
-  (2015) ActivityNet: a large-scale video benchmark for human activity understanding. Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961–970. Cited by: §3.1.
-  (2017) Localizing moments in video with natural language. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5804–5813. Cited by: §2.
-  (1997) Long short-term memory. Neural Computation 9, pp. 1735–1780. Cited by: §4.1.1.
-  (2016) Attention-based multimodal neural machine translation. In Proceedings of the 1st Conference on Machine Translation (WMT), pp. 639–645. Cited by: §2.
-  (2017) The kinetics human action video dataset. arXiv preprint arXiv:1705.06950. Cited by: Appendix A, §3.1, §4.1.2.
-  (2015) Adam: a method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations (ICLR), Cited by: Appendix A, Appendix A.
-  (2017) Dense-captioning events in videos. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), pp. 706–715. Cited by: Table 1, §2.
-  (2011) HMDB: a large video database for human motion recognition. Proceedings of the 2011 International Conference on Computer Vision (ICCV), pp. 2556–2563. Cited by: §3.1.
-  (2017) Fluency-guided cross-lingual image captioning. In Proceedings of the 25th ACM international conference on Multimedia (ACM-MM), Cited by: §2.
-  (2019) COCO-cn for cross-lingual image tagging, captioning and retrieval. IEEE Transactions on Multimedia. Cited by: §2.
-  (2016) TGIF: a new dataset and benchmark on animated gif description. In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4641–4650. Cited by: Table 1.
-  (2017) Attention strategies for multi-source sequence-to-sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 196–202. Cited by: §2.
-  (2004) ROUGE: a package for automatic evaluation of summaries. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), Cited by: §4.1.2.
-  (2014) Visual semantic search: retrieving videos via complex textual queries. Proceedings of the 27th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2657–2664. Cited by: §5.
-  (2015) Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1412–1421. Cited by: §4.2.1.
-  (2017) OSU multimodal machine translation system report. In Proceedings of the 2nd Conference on Machine Translation (WMT), pp. 465–469. Cited by: §2.
-  (2017) Sheffield multimt: using object posterior predictions for multimodal machine translation. In Proceedings of the 2nd Conference on Machine Translation (WMT), pp. 470–476. Cited by: §2.
-  (2001) Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics(ACL), Cited by: §4.1.2.
-  (2016) Multilingual visual sentiment concept matching. In Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval (ICMR), Cited by: §2.
-  (2017) Reinforced video captioning with entailment rewards. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §4.1.1.
-  (2015) Flickr30k entities: collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the 2015 IEEE international conference on computer vision (ICCV), pp. 2641–2649. Cited by: §2.
-  (2013) Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics (TACL) 1, pp. 25–36. Cited by: Table 1, §2.
-  (2014) Coherent multi-sentence video description with variable level of detail. In Proceedings of the German Conference on Pattern Recognition, pp. 184–195. Cited by: Table 1, §2.
-  (2015) A dataset for movie description. In Proceedings of the 28th IEEE conference on computer vision and pattern recognition (CVPR), pp. 3202–3212. Cited by: §1, Table 1, §2.
-  (2017-05-01) Movie description. International Journal of Computer Vision (IJCV) 123 (1), pp. 94–120. Cited by: Table 1, §2.
-  (2018) How2: a large-scale dataset for multimodal language understanding. In Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL), Cited by: §2.
-  (1997-11) Bidirectional recurrent neural networks. Trans. Sig. Proc. 45 (11), pp. 2673–2681. External Links: Cited by: §4.1.1.
-  (2018) Visual question answering dataset for bilingual image understanding: a study of cross-lingual transfer using attention maps. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), Cited by: §2.
-  (2016) Hollywood in homes: crowdsourcing data collection for activity understanding. In Proceedings of the 2016 European Conference on Computer Vision (ECCV), pp. 510–526. Cited by: Table 1, §2.
-  (2012) UCF101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. Cited by: §3.1.
-  (2016) A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the 1st Conference on Machine Translation (WMT), pp. 543–553. Cited by: §2.
-  (2014) Sequence to sequence learning with neural networks. In Proceedings of the 28th Conference on Neural Information Processing Systems (NeurIPS), pp. 3104–3112. Cited by: §4.2.1.
-  (2015) Using descriptive video services to create a large data source for video annotation research. arXiv preprint arXiv:1503.01070. Cited by: Table 1, §2.
-  (2017) Using artificial tokens to control languages for multilingual image caption generation. arXiv preprint arXiv:1706.06275. Cited by: §2.
-  (2015) CIDEr: consensus-based image description evaluation. Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4566–4575. Cited by: §4.1.2.
-  (2015) Sequence to sequence-video to text. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4534–4542. Cited by: §4.2.1.
-  (2015) Show and tell: a neural image caption generator. Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3156–3164. Cited by: §2.
-  (2016) Actions transformations. Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2658–2667. Cited by: §3.1.
-  (2018) Video captioning via hierarchical reinforcement learning. In Proceedings of the 31st IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §4.1.1.
-  (2018) Watch, listen, and describe: globally and locally aligned cross-modal attentions for video captioning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), Cited by: §4.1.3.
-  (2019) Learning to compose topic-aware mixture of experts for zero-shot video captioning. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI), Cited by: §5.
-  (2000) The part-of-speech tagging guidelines for the penn chinese treebank (3.0). IRCS Technical Reports Series, pp. 38. Cited by: Table 3.
-  (2016) MSR-vtt: a large video description dataset for bridging video and language. In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research vatex-challenge.org, §1, §1, Table 1, §2, §3.2.
-  (1990) Measuring lexical style and competence: the type-token vocabulary curve. Style 24 (4), pp. 584–599. Cited by: §3.2.
-  (2016) Building an arabic machine translation post-edited corpus: guidelines and annotation. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC), pp. 1869–1876. Cited by: §3.1.2.
-  (2016) Generation for user generated videos. In Proceedings of the 2016 European conference on computer vision (ECCV), pp. 609–625. Cited by: Table 1, §2.
-  (2019) Man: moment alignment network for natural language moment retrieval via iterative graph adjustment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1247–1257. Cited by: §5.
-  (2016) Sadness expressions in english and chinese: corpus linguistic contrastive semantic analysis. Bloomsbury Publishing. Cited by: Table 3.
-  (2018) Towards automatic learning of procedures from web instructional videos. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI), Cited by: §1, Table 1, §2.
Appendix A Implementation Details
Multilingual Video Captioning
To preprocess the videos, we sample each video at and extract the I3D features  from these sampled frames. The I3D model is pretrained on the original Kinetics training dataset  and used here without fine-tuning. Both the English and Chinese captions are truncated to a maximum of 30 words. Note that we use the segmented Chinese words55 5 We use the open-source tool Jieba for Chinese word segmentation: https://github.com/fxsjy/jieba rather than raw Chinese characters. The vocabularies are built with a minimum word count , resulting in around English words and about Chinese words.
All the hyperparameters are tuned on the validation sets and same for both English and Chinese caption training. The video encoder is a bi-LSTM of size and the decoder LSTM is of size . The dimensions of the word embedding layers are . All models are trained using MLE loss and optimized using Adam optimizer  with a batch size . We adopt Dropout for regularization. The learning rate is initially set as and then halted when the current CIDEr score does not surpass the previous best for epochs. Schedule sampling  is employed to train the models. The probability of schedule sampling is first set to be 0.05, then increased by 0.05 every 5 epochs, and eventually fixed at 0.25 after 25 epoches. At test time, we use beam search of size to report the final results.
Video-guided Machine Translation
The data prepossessing steps are the same as above except that we truncate the captions with a maximum length of here. The baseline NMT is composed of a -layer bi-LSTM encoder of size and a -layer LSTM decoder of size . The dimensions of both English and Chinese word embeddings are . The video encoder is a bi-LSTM of size . MLE loss is implemented to train the model using Adam optimizer . The batch size is during training and early-stopping is used to choose the models. Then we fine-tune the private parameters of model for each language with the shared parameters fixed. As for evaluation, we use beam search of size to report the results on the BLEU-4 metric.
Appendix B Data Collection Interfaces
We show the AMT interface for English caption collection in Figure 8. Since the Chinese captions are divided into two parts, we build two separate interfaces, one of which is to collect the captions that directly describe the video (Figure 9) and the other for collecting the Chinese translations parallel to the English captions (Figure 10).
Appendix C More VaTeX Samples
In addition to the example shown in the main paper, Figure 11 demonstrates more samples of our VaTeX dataset.
Appendix D Qualitative Results
Multilingual Video Captioning
Figure 12 illustrates some qualitative examples of multilingual video captioning, where we compare both the English and Chinese results generated by the monolingual models (Base), the multilingual model that shares the video encoder for English & Chinese (Shared Enc), and the multilingual model that shares both the video encoder and the language decoder for English & Chinese (Shared Enc-Dec).