Abstract
We propose a novel memory network model named Read-Write Memory Network(RWMN) to perform question and answering tasks for large-scale, multimodalmovie story understanding. The key focus of our RWMN model is to design theread network and the write network that consist of multiple convolutionallayers, which enable memory read and write operations to have high capacity andflexibility. While existing memory-augmented network models treat each memoryslot as an independent block, our use of multi-layered CNNs allows the model toread and write sequential memory cells as chunks, which is more reasonable torepresent a sequential story because adjacent memory blocks often have strongcorrelations. For evaluation, we apply our model to all the six tasks of theMovieQA benchmark, and achieve the best accuracies on several tasks, especiallyon the visual QA task. Our model shows a potential to better understand notonly the content in the story, but also more abstract information, such asrelationships between characters and the reasons for their actions.