Abstract
Robot models, particularly those trained with large amounts of data, haverecently shown a plethora of real-world manipulation and navigationcapabilities. Several independent efforts have shown that given sufficienttraining data in an environment, robot policies can generalize to demonstratedvariations in that environment. However, needing to finetune robot models toevery new environment stands in stark contrast to models in language or visionthat can be deployed zero-shot for open-world problems. In this work, wepresent Robot Utility Models (RUMs), a framework for training and deployingzero-shot robot policies that can directly generalize to new environmentswithout any finetuning. To create RUMs efficiently, we develop new tools toquickly collect data for mobile manipulation tasks, integrate such data into apolicy with multi-modal imitation learning, and deploy policies on-device onHello Robot Stretch, a cheap commodity robot, with an external mLLM verifierfor retrying. We train five such utility models for opening cabinet doors,opening drawers, picking up napkins, picking up paper bags, and reorientingfallen objects. Our system, on average, achieves 90% success rate in unseen,novel environments interacting with unseen objects. Moreover, the utilitymodels can also succeed in different robot and camera set-ups with no furtherdata, training, or fine-tuning. Primary among our lessons are the importance oftraining data over training algorithm and policy class, guidance about datascaling, necessity for diverse yet high-quality demonstrations, and a recipefor robot introspection and retrying to improve performance on individualenvironments. Our code, data, models, hardware designs, as well as ourexperiment and deployment videos are open sourced and can be found on ourproject website: https://robotutilitymodels.com