Abstract
The rapid advancement of Multimodal Large Language Models (MLLMs) has madealigning them with human preferences a critical challenge. Reward Models (RMs)are a core technology for achieving this goal, but a systematic guide forbuilding state-of-the-art Multimodal Reward Models (MRMs) is currently lackingin both academia and industry. Through exhaustive experimental analysis, thispaper aims to provide a clear ``recipe'' for constructing high-performanceMRMs. We systematically investigate every crucial component in the MRMdevelopment pipeline, including \textit{reward modeling paradigms} (e.g.,Naive-RM, Critic-based RM, and Generative RM), \textit{reward headarchitecture}, \textit{training strategies}, \textit{data curation} (coveringover ten multimodal and text-only preference datasets), \textit{backbone model}and \textit{model scale}, and \textit{ensemble methods}. Based on these experimental insights, we introduce \textbf{BaseReward}, apowerful and efficient baseline for multimodal reward modeling. BaseRewardadopts a simple yet effective architecture, built upon a {Qwen2.5-VL} backbone,featuring an optimized two-layer reward head, and is trained on a carefullycurated mixture of high-quality multimodal and text-only preference data. Ourresults show that BaseReward establishes a new SOTA on major benchmarks such asMM-RLHF-Reward Bench, VL-Reward Bench, and Multimodal Reward Bench,outperforming previous models. Furthermore, to validate its practical utilitybeyond static benchmarks, we integrate BaseReward into a real-worldreinforcement learning pipeline, successfully enhancing an MLLM's performanceacross various perception, reasoning, and conversational tasks. This work notonly delivers a top-tier MRM but, more importantly, provides the community witha clear, empirically-backed guide for developing robust reward models for thenext generation of MLLMs.