Abstract
Throughout history, we have successfully integrated various machines into ourhomes. Dishwashers, laundry machines, stand mixers, and robot vacuums are a fewrecent examples. However, these machines excel at performing only a single taskeffectively. The concept of a "generalist machine" in homes - a domesticassistant that can adapt and learn from our needs, all while remainingcost-effective - has long been a goal in robotics that has been steadilypursued for decades. In this work, we initiate a large-scale effort towardsthis goal by introducing Dobb-E, an affordable yet versatile general-purposesystem for learning robotic manipulation within household settings. Dobb-E canlearn a new task with only five minutes of a user showing it how to do it,thanks to a demonstration collection tool ("The Stick") we built out of cheapparts and iPhones. We use the Stick to collect 13 hours of data in 22 homes ofNew York City, and train Home Pretrained Representations (HPR). Then, in anovel home environment, with five minutes of demonstrations and fifteen minutesof adapting the HPR model, we show that Dobb-E can reliably solve the task onthe Stretch, a mobile robot readily available on the market. Across roughly 30days of experimentation in homes of New York City and surrounding areas, wetest our system in 10 homes, with a total of 109 tasks in differentenvironments, and finally achieve a success rate of 81%. Beyond successpercentages, our experiments reveal a plethora of unique challenges absent orignored in lab robotics. These range from effects of strong shadows, tovariable demonstration quality by non-expert users. With the hope ofaccelerating research on home robots, and eventually seeing robot butlers inevery home, we open-source Dobb-E software stack and models, our data, and ourhardware designs at https://dobb-e.com