Abstract
Recent advances in reinforcement learning (RL) have strengthened thereasoning capabilities of vision-language models (VLMs). However, enhancingpolicy exploration to better scale test-time compute remains largelyunderexplored. In addition, VLMs continue to struggle with imperfect visualperception, which in turn affects the subsequent reasoning process. To thisend, we propose NoisyRollout, a simple yet effective data augmentation methodthat mixes trajectories from both clean and moderately distorted images duringRL training. By injecting targeted diversity in visual perception and theresulting reasoning patterns, NoisyRollout promotes better policy explorationthrough vision-oriented inductive biases, ultimately leading to more robustreasoning behaviors. We further adopt a noise annealing schedule that graduallyreduces distortion strength over training, leveraging noisy signals early onwhile ensuring training stability in later stages. Crucially, our method iseasy-to-adopt--requiring no additional training cost and no modifications tothe RL objective. Extensive experiments on $2$ distinct training datasetsdemonstrate that NoisyRollout achieves state-of-the-art performance amongopen-source RL-tuned models across $5$ out-of-domain reasoning and perceptionbenchmarks. Furthermore, we validate the effectiveness of NoisyRollout acrossmodel sizes ($7$B and $32$B) and data scales (from $1$K to $6$K), highlightingits generalizability and scalability.