Abstract
In this paper, we study the problem of mapping natural language instructionsto complex spatial actions in a 3D blocks world. We first introduce a newdataset that pairs complex 3D spatial operations to rich natural languagedescriptions that require complex spatial and pragmatic interpretations such as"mirroring", "twisting", and "balancing". This dataset, built on the simulationenvironment of Bisk, Yuret, and Marcu (2016), attains language that issignificantly richer and more complex, while also doubling the size of theoriginal dataset in the 2D environment with 100 new world configurations and250,000 tokens. In addition, we propose a new neural architecture that achievescompetitive results while automatically discovering an inventory ofinterpretable spatial operations (Figure 5)