Abstract
Reward models (RMs) have become essential for aligning large language models(LLMs), serving as scalable proxies for human evaluation in both training andinference. However, existing RMs struggle on knowledge-intensive and long-formtasks, where evaluating correctness requires grounding beyond the model'sinternal knowledge. This limitation hinders them from reliably discriminatingsubtle quality differences, especially when external evidence is necessary. Toaddress this, we introduce OpenRM, a tool-augmented long-form reward model thatsystematically judges open-ended responses by invoking external tools to gatherrelevant evidence. We train OpenRM with Group Relative Policy Optimization(GRPO) on over 27K synthesized pairwise examples generated through acontrollable data synthesis framework. The training objective jointlysupervises intermediate tool usage and final outcome accuracy, incentivizingour reward model to learn effective evidence-based judgment strategies.Extensive experiments on three newly-collected datasets and two widely-usedbenchmarks demonstrate that OpenRM substantially outperforms existing rewardmodeling approaches. As a further step, we integrate OpenRM into bothinference-time response selection and training-time data selection. This yieldsconsistent gains in downstream LLM alignment tasks, highlighting the potentialof tool-augmented reward models for scaling reliable long-form evaluation.