Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged asa powerful paradigm for post-training large language models (LLMs), achievingstate-of-the-art performance on tasks with structured, verifiable answers.Applying RLVR to Multimodal LLMs (MLLMs) presents significant opportunities butis complicated by the broader, heterogeneous nature of vision-language tasksthat demand nuanced visual, logical, and spatial capabilities. As such,training MLLMs using RLVR on multiple datasets could be beneficial but createschallenges with conflicting objectives from interaction among diverse datasets,highlighting the need for optimal dataset mixture strategies to improvegeneralization and reasoning. We introduce a systematic post-training frameworkfor Multimodal LLM RLVR, featuring a rigorous data mixture problem formulationand benchmark implementation. Specifically, (1) We developed a multimodal RLVRframework for multi-dataset post-training by curating a dataset that containsdifferent verifiable vision-language problems and enabling multi-domain onlineRL learning with different verifiable rewards; (2) We proposed a data mixturestrategy that learns to predict the RL fine-tuning outcome from the datamixture distribution, and consequently optimizes the best mixture.Comprehensive experiments showcase that multi-domain RLVR training, whencombined with mixture prediction strategies, can significantly boost MLLMgeneral reasoning capacities. Our best mixture improves the post-trainedmodel's accuracy on out-of-distribution benchmarks by an average of 5.24%compared to the same model post-trained with uniform data mixture, and by atotal of 20.74% compared to the pre-finetuning baseline.