Abstract
The reasoning capabilities of large language models (LLMs) have advancedrapidly, particularly following the release of DeepSeek R1, which has inspireda surge of research into data quality and reinforcement learning (RL)algorithms. Despite the pivotal role diversity plays in RL, its influence onLLM reasoning remains largely underexplored. To bridge this gap, this workpresents a systematic investigation into the impact of diversity in RL-basedtraining for LLM reasoning, and proposes a novel diversity-aware policyoptimization method. Across evaluations on 12 LLMs, we observe a strongpositive correlation between the solution diversity and Potential at k (a novelmetric quantifying an LLM's reasoning potential) in high-performing models.This finding motivates our method to explicitly promote diversity during RLtraining. Specifically, we design a token-level diversity and reformulate itinto a practical objective, then we selectively apply it to positive samples.Integrated into the R1-zero training framework, our method achieves a 3.5percent average improvement across four mathematical reasoning benchmarks,while generating more diverse and robust solutions.