Abstract
We have a vision of a day when autonomous robots can collaborate with humansas assistants in performing complex tasks in the physical world. This visionincludes that the robots will have the ability to communicate with their humancollaborators using language that is natural to the humans. TraditionalInteractive Task Learning (ITL) systems have some of this ability, but thelanguage they can understand is very limited. The advent of large languagemodels (LLMs) provides an opportunity to greatly improve the languageunderstanding of robots, yet integrating the language abilities of LLMs withrobots that operate in the real physical world is a challenging problem. In this chapter we first review briefly a few commercial robot products thatwork closely with humans, and discuss how they could be much bettercollaborators with robust language abilities. We then explore how an AI systemwith a cognitive agent that controls a physical robot at its core, interactswith both a human and an LLM, and accumulates situational knowledge through itsexperiences, can be a possible approach to reach that vision. We focus on threespecific challenges of having the robot understand natural language, andpresent a simple proof-of-concept experiment using ChatGPT for each. Finally,we discuss what it will take to turn these simple experiments into anoperational system where LLM-assisted language understanding is a part of anintegrated robotic assistant that uses language to collaborate with humans.