Abstract
Recent years have witnessed an increasing interest in image-basedquestion-answering (QA) tasks. However, due to data limitations, there has beenmuch less work on video-based QA. In this paper, we present TVQA, a large-scalevideo QA dataset based on 6 popular TV shows. TVQA consists of 152,545 QA pairsfrom 21,793 clips, spanning over 460 hours of video. Questions are designed tobe compositional in nature, requiring systems to jointly localize relevantmoments within a clip, comprehend subtitle-based dialogue, and recognizerelevant visual concepts. We provide analyses of this new dataset as well asseveral baselines and a multi-stream end-to-end trainable neural networkframework for the TVQA task. The dataset is publicly available athttp://tvqa.cs.unc.edu.