Abstract
Offline preference-based reinforcement learning (PbRL) mitigates the need forreward definition, aligning with human preferences via preference-driven rewardfeedback without interacting with the environment. However, the effectivenessof preference-driven reward functions depends on the modeling ability of thelearning model, which current MLP-based and Transformer-based methods may failto adequately provide. To alleviate the failure of the reward function causedby insufficient modeling, we propose a novel preference-based rewardacquisition method: Diffusion Preference-based Reward (DPR). Unlike previousmethods using Bradley-Terry models for trajectory preferences, we use diffusionmodels to directly model preference distributions for state-action pairs,allowing rewards to be discriminatively obtained from these distributions. Inaddition, considering the particularity of preference data that only know theinternal relationships of paired trajectories, we further propose ConditionalDiffusion Preference-based Reward (C-DPR), which leverages relative preferenceinformation to enhance the construction of the diffusion model. We apply theabove methods to existing offline reinforcement learning algorithms and aseries of experiment results demonstrate that the diffusion-based rewardacquisition approach outperforms previous MLP-based and Transformer-basedmethods.