Abstract
Natural language is one of the most intuitive ways to express human intent.However, translating instructions and commands towards robotic motiongeneration, and deployment in the real world, is far from being an easy task.Indeed, combining robotic's inherent low-level geometric and kinodynamicconstraints with human's high-level semantic information reinvigorates andraises new challenges to the task-design problem -- typically leading to taskor hardware specific solutions with a static set of action targets andcommands. This work instead proposes a flexible language-based framework thatallows to modify generic 3D robotic trajectories using language commands withreduced constraints about prior task or robot information. By taking advantageof pre-trained language models, we employ an auto-regressive transformer to mapnatural language inputs and contextual images into changes in 3D trajectories.We show through simulations and real-life experiments that the model cansuccessfully follow human intent, modifying the shape and speed of trajectoriesfor multiple robotic platforms and contexts. This study takes a step intobuilding large pre-trained foundational models for robotics and shows how suchmodels can create more intuitive and flexible interactions between human andmachines. Codebase available at:https://github.com/arthurfenderbucker/NL_trajectory_reshaper.