Abstract
Enabling robots to understand instructions provided via spoken naturallanguage would facilitate interaction between robots and people in a variety ofsettings in homes and workplaces. However, natural language instructions areoften missing information that would be obvious to a human based onenvironmental context and common sense, and hence does not need to beexplicitly stated. In this paper, we introduce Language-Model-based CommonsenseReasoning (LMCR), a new method which enables a robot to listen to a naturallanguage instruction from a human, observe the environment around it, andautomatically fill in information missing from the instruction usingenvironmental context and a new commonsense reasoning approach. Our approachfirst converts an instruction provided as unconstrained natural language into aform that a robot can understand by parsing it into verb frames. Our approachthen fills in missing information in the instruction by observing objects inits vicinity and leveraging commonsense reasoning. To learn commonsensereasoning automatically, our approach distills knowledge from largeunstructured textual corpora by training a language model. Our results show thefeasibility of a robot learning commonsense knowledge automatically fromweb-based textual corpora, and the power of learned commonsense reasoningmodels in enabling a robot to autonomously perform tasks based on incompletenatural language instructions.