Abstract
Reinforcement learning (RL) trains an agent from experiences interacting withthe environment. In scenarios where online interactions are impractical,offline RL, which trains the agent using pre-collected datasets, has becomepopular. While this new paradigm presents remarkable effectiveness acrossvarious real-world domains, like healthcare and energy management, there is agrowing demand to enable agents to rapidly and completely eliminate theinfluence of specific trajectories from both the training dataset and thetrained agents. To meet this problem, this paper advocates Trajdeleter, thefirst practical approach to trajectory unlearning for offline RL agents. Thekey idea of Trajdeleter is to guide the agent to demonstrate deterioratingperformance when it encounters states associated with unlearning trajectories.Simultaneously, it ensures the agent maintains its original performance levelwhen facing other remaining trajectories. Additionally, we introduceTrajauditor, a simple yet efficient method to evaluate whether Trajdeletersuccessfully eliminates the specific trajectories of influence from the offlineRL agent. Extensive experiments conducted on six offline RL algorithms andthree tasks demonstrate that Trajdeleter requires only about 1.5% of the timeneeded for retraining from scratch. It effectively unlearns an average of 94.8%of the targeted trajectories yet still performs well in actual environmentinteractions after unlearning. The replication package and agent parameters areavailable online.