Abstract
The canonical approach to video-and-language learning (e.g., video questionanswering) dictates a neural model to learn from offline-extracted dense videofeatures from vision models and text features from language models. Thesefeature extractors are trained independently and usually on tasks differentfrom the target domains, rendering these fixed features sub-optimal fordownstream tasks. Moreover, due to the high computational overload of densevideo features, it is often difficult (or infeasible) to plug featureextractors directly into existing approaches for easy finetuning. To provide aremedy to this dilemma, we propose a generic framework ClipBERT that enablesaffordable end-to-end learning for video-and-language tasks, by employingsparse sampling, where only a single or a few sparsely sampled short clips froma video are used at each training step. Experiments on text-to-video retrievaland video question answering on six datasets demonstrate that ClipBERToutperforms (or is on par with) existing methods that exploit full-lengthvideos, suggesting that end-to-end learning with just a few sparsely sampledclips is often more accurate than using densely extracted offline features fromfull-length videos, proving the proverbial less-is-more principle. Videos inthe datasets are from considerably different domains and lengths, ranging from3-second generic domain GIF videos to 180-second YouTube human activity videos,showing the generalization ability of our approach. Comprehensive ablationstudies and thorough analyses are provided to dissect what factors lead to thissuccess. Our code is publicly available at https://github.com/jayleicn/ClipBERT