Abstract
Generalist robot policies trained on large-scale datasets such as OpenX-Embodiment (OXE) demonstrate strong performance across a wide range of tasks.However, they often struggle to generalize beyond the distribution of theirtraining data. In this paper, we investigate the underlying cause of thislimited generalization capability. We identify shortcut learning -- thereliance on task-irrelevant features -- as a key impediment to generalization.Through comprehensive theoretical and empirical analysis, we uncover twoprimary contributors to shortcut learning: (1) limited diversity withinindividual sub-datasets, and (2) significant distributional disparities acrosssub-datasets, leading to dataset fragmentation. These issues arise from theinherent structure of large-scale datasets like OXE, which are typicallycomposed of multiple sub-datasets collected independently across variedenvironments and embodiments. Our findings provide critical insights intodataset collection strategies that can reduce shortcut learning and enhance thegeneralization ability of generalist robot policies. Moreover, in scenarioswhere acquiring new large-scale data is impractical, we demonstrate thatcarefully selected robotic data augmentation strategies can effectively reduceshortcut learning in existing offline datasets, thereby improvinggeneralization capabilities of generalist robot policies, e.g., $\pi_0$, inboth simulation and real-world environments. More information athttps://lucky-light-sun.github.io/proj/shortcut-learning-in-grps/.