Abstract
Effective collaboration of dual-arm robots and their tool use capabilitiesare increasingly important areas in the advancement of robotics. These skillsplay a significant role in expanding robots' ability to operate in diversereal-world environments. However, progress is impeded by the scarcity ofspecialized training data. This paper introduces RoboTwin, a novel benchmarkdataset combining real-world teleoperated data with synthetic data from digitaltwins, designed for dual-arm robotic scenarios. Using the COBOT Magic platform,we have collected diverse data on tool usage and human-robot interaction. Wepresent a innovative approach to creating digital twins using AI-generatedcontent, transforming 2D images into detailed 3D models. Furthermore, weutilize large language models to generate expert-level training data andtask-specific pose sequences oriented toward functionality. Our keycontributions are: 1) the RoboTwin benchmark dataset, 2) an efficientreal-to-simulation pipeline, and 3) the use of language models for automaticexpert-level data generation. These advancements are designed to address theshortage of robotic training data, potentially accelerating the development ofmore capable and versatile robotic systems for a wide range of real-worldapplications. The project page is available athttps://robotwin-benchmark.github.io/early-version/