Abstract
Generalization remains one of the most important desiderata for robust robotlearning systems. While recently proposed approaches show promise ingeneralization to novel objects, semantic concepts, or visual distributionshifts, generalization to new tasks remains challenging. For example, alanguage-conditioned policy trained on pick-and-place tasks will not be able togeneralize to a folding task, even if the arm trajectory of folding is similarto pick-and-place. Our key insight is that this kind of generalization becomesfeasible if we represent the task through rough trajectory sketches. We proposea policy conditioning method using such rough trajectory sketches, which wecall RT-Trajectory, that is practical, easy to specify, and allows the policyto effectively perform new tasks that would otherwise be challenging toperform. We find that trajectory sketches strike a balance between beingdetailed enough to express low-level motion-centric guidance while being coarseenough to allow the learned policy to interpret the trajectory sketch in thecontext of situational visual observations. In addition, we show how trajectorysketches can provide a useful interface to communicate with robotic policies:they can be specified through simple human inputs like drawings or videos, orthrough automated methods such as modern image-generating orwaypoint-generating methods. We evaluate RT-Trajectory at scale on a variety ofreal-world robotic tasks, and find that RT-Trajectory is able to perform awider range of tasks compared to language-conditioned and goal-conditionedpolicies, when provided the same training data.