Abstract
Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learningwith Verifiable Rewards (RLVR) are the main RL paradigms used in LLMpost-training, each offering distinct advantages. However, RLHF struggles withinterpretability and reward hacking because it relies on human judgments thatusually lack explicit criteria, whereas RLVR is limited in scope by its focuson correctness-based verifiers. We propose Reinforcement Learning with BinaryFlexible Feedback (RLBFF), which combines the versatility of human-drivenpreferences with the precision of rule-based verification, enabling rewardmodels to capture nuanced aspects of response quality beyond mere correctness.RLBFF extracts principles that can be answered in a binary fashion (e.g.accuracy of information: yes, or code readability: no) from natural languagefeedback. Such principles can then be used to ground Reward Model training asan entailment task (response satisfies or does not satisfy an arbitraryprinciple). We show that Reward Models trained in this manner can outperformBradley-Terry models when matched for data and achieve top performance onRM-Bench (86.2%) and JudgeBench (81.4%, #1 on leaderboard as of September 24,2025). Additionally, users can specify principles of interest at inference timeto customize the focus of our reward models, in contrast to Bradley-Terrymodels. Finally, we present a fully open source recipe (including data) toalign Qwen3-32B using RLBFF and our Reward Model, to match or exceed theperformance of o3-mini and DeepSeek R1 on general alignment benchmarks ofMT-Bench, WildBench, and Arena Hard v2 (at <5% of the inference cost). Models:https://huggingface.co/collections/nvidia/reward-models-10-2025