Abstract
Controlling robots to perform tasks via natural language is one of the mostchallenging topics in human-robot interaction. In this work, we present a robotsystem that follows unconstrained language instructions to pick and placearbitrary objects and effectively resolves ambiguities through dialogues. Ourapproach infers objects and their relationships from input images and languageexpressions and can place objects in accordance with the spatial relationsexpressed by the user. Unlike previous approaches, we consider grounding notonly for the picking but also for the placement of everyday objects fromlanguage. Specifically, by grounding objects and their spatial relations, weallow specification of complex placement instructions, e.g. "place it behindthe middle red bowl". Our results obtained using a real-world PR2 robotdemonstrate the effectiveness of our method in understanding pick-and-placelanguage instructions and sequentially composing them to solve tabletopmanipulation tasks. Videos are available athttp://speechrobot.cs.uni-freiburg.de