Abstract
Direct alignment methods are increasingly used for aligning large languagemodels (LLMs) with human preferences. However, these methods suffer from theissues of verbosity and likelihood displacement, which can be driven by thenoisy preference pairs that induce similar likelihood for preferred anddispreferred responses. The contributions of this paper are two-fold. First, wepropose a new preference alignment method based on comparison oracles andprovide the convergence guarantee for its basic scheme. Second, we improve ourmethod using some heuristics and conduct the experiments to demonstrate theflexibility and compatibility of practical scheme in improving the performanceof LLMs using noisy preference pairs. Evaluations are conducted across multiplebase and instruction-tuned models (Mistral-7B, Llama-3-8B and Gemma-2-9B) withbenchmarks (AlpacaEval 2, MT-Bench and Arena-Hard). Experimental results showthe effectiveness of our method as an alternative to addressing the limitationsof existing direct alignment methods. A highlight of our work is that weevidence the importance of designing specialized methods for preference pairswith distinct likelihood margin, which complements the recent findings in\citet{Razin-2025-Unintentional}.