Abstract
Recent developments in Large Language Models pre-trained on extensive corporahave shown significant success in various natural language processing taskswith minimal fine-tuning. This success offers new promise for robotics, whichhas long been constrained by the high cost of action-labeled data. We ask:given the abundant video data containing interaction-related knowledgeavailable as a rich "corpus", can a similar generative pre-training approach beeffectively applied to enhance robot learning? The key challenge is to identifyan effective representation for autoregressive pre-training that benefits robotmanipulation tasks. Inspired by the way humans learn new skills throughobserving dynamic environments, we propose that effective robotic learningshould emphasize motion-related knowledge, which is closely tied to low-levelactions and is hardware-agnostic, facilitating the transfer of learned motionsto actual robot actions. To this end, we introduce Moto, which converts videocontent into latent Motion Token sequences by a Latent Motion Tokenizer,learning a bridging "language" of motion from videos in an unsupervised manner.We pre-train Moto-GPT through motion token autoregression, enabling it tocapture diverse visual motion knowledge. After pre-training, Moto-GPTdemonstrates the promising ability to produce semantically interpretable motiontokens, predict plausible motion trajectories, and assess trajectoryrationality through output likelihood. To transfer learned motion priors toreal robot actions, we implement a co-fine-tuning strategy that seamlesslybridges latent motion token prediction and real robot control. Extensiveexperiments show that the fine-tuned Moto-GPT exhibits superior robustness andefficiency on robot manipulation benchmarks, underscoring its effectiveness intransferring knowledge from video data to downstream visual manipulation tasks.