Abstract
Recent breakthroughs in computer vision and natural language processing havespurred interest in challenging multi-modal tasks such as visualquestion-answering and visual dialogue. For such tasks, one successful approachis to condition image-based convolutional network computation on language viaFeature-wise Linear Modulation (FiLM) layers, i.e., per-channel scaling andshifting. We propose to generate the parameters of FiLM layers going up thehierarchy of a convolutional network in a multi-hop fashion rather than all atonce, as in prior work. By alternating between attending to the language inputand generating FiLM layer parameters, this approach is better able to scale tosettings with longer input sequences such as dialogue. We demonstrate thatmulti-hop FiLM generation achieves state-of-the-art for the short inputsequence task ReferIt --- on-par with single-hop FiLM generation --- while alsosignificantly outperforming prior state-of-the-art and single-hop FiLMgeneration on the GuessWhat?! visual dialogue task.