Abstract
Although virtual agents are increasingly situated in environments wherenatural language is the most effective mode of interaction with humans, theseexchanges are rarely used as an opportunity for learning. Leveraging languageinteractions effectively requires addressing limitations in the two most commonapproaches to language grounding: semantic parsers built on top of fixed objectcategories are precise but inflexible and end-to-end models are maximallyexpressive, but fickle and opaque. Our goal is to develop a system thatbalances the strengths of each approach so that users can teach agents newinstructions that generalize broadly from a single example. We introduce theidea of neural abstructions: a set of constraints on the inference procedure ofa label-conditioned generative model that can affect the meaning of the labelin context. Starting from a core programming language that operates overabstructions, users can define increasingly complex mappings from naturallanguage to actions. We show that with this method a user population is able tobuild a semantic parser for an open-ended house modification task in Minecraft.The semantic parser that results is both flexible and expressive: thepercentage of utterances sourced from redefinitions increases steadily over thecourse of 191 total exchanges, achieving a final value of 28%.