Abstract
We introduce the Neural State Machine, seeking to bridge the gap between theneural and symbolic views of AI and integrate their complementary strengths forthe task of visual reasoning. Given an image, we first predict a probabilisticgraph that represents its underlying semantics and serves as a structured worldmodel. Then, we perform sequential reasoning over the graph, iterativelytraversing its nodes to answer a given question or draw a new inference. Incontrast to most neural architectures that are designed to closely interactwith the raw sensory data, our model operates instead in an abstract latentspace, by transforming both the visual and linguistic modalities into semanticconcept-based representations, thereby achieving enhanced transparency andmodularity. We evaluate our model on VQA-CP and GQA, two recent VQA datasetsthat involve compositionality, multi-step inference and diverse reasoningskills, achieving state-of-the-art results in both cases. We provide furtherexperiments that illustrate the model's strong generalization capacity acrossmultiple dimensions, including novel compositions of concepts, changes in theanswer distribution, and unseen linguistic structures, demonstrating thequalities and efficacy of our approach.