Abstract
Vision and language understanding has emerged as a subject undergoing intensestudy in Artificial Intelligence. Among many tasks in this line of research,visual question answering (VQA) has been one of the most successful ones, wherethe goal is to learn a model that understands visual content at region-leveldetails and finds their associations with pairs of questions and answers in thenatural language form. Despite the rapid progress in the past few years, mostexisting work in VQA have focused primarily on images. In this paper, we focuson extending VQA to the video domain and contribute to the literature in threeimportant ways. First, we propose three new tasks designed specifically forvideo VQA, which require spatio-temporal reasoning from videos to answerquestions correctly. Next, we introduce a new large-scale dataset for video VQAnamed TGIF-QA that extends existing VQA work with our new tasks. Finally, wepropose a dual-LSTM based approach with both spatial and temporal attention,and show its effectiveness over conventional VQA techniques through empiricalevaluations.