Offline reinforcement learning (RL) aims at learning policies from previouslycollected static trajectory data without interacting with the real environment.Recent works provide a novel perspective by viewing offline RL as a genericsequence generation problem, adopting sequence models such as Transformerarchitecture to model distributions over trajectories, and repurposing beamsearch as a planning algorithm. However, the training datasets utilized ingeneral offline RL tasks are quite limited and often suffer from insufficientdistribution coverage, which could be harmful to training sequence generationmodels yet has not drawn enough attention in the previous works. In this paper,we propose a novel algorithm named Bootstrapped Transformer, which incorporatesthe idea of bootstrapping and leverages the learned model to self-generate moreoffline data to further boost the sequence model training. We conduct extensiveexperiments on two offline RL benchmarks and demonstrate that our model canlargely remedy the existing offline RL training limitations and beat otherstrong baseline methods. We also analyze the generated pseudo data and therevealed characteristics may shed some light on offline RL training. The codesare available at https://seqml.github.io/bootorl.