Abstract
Data selection is a critical aspect of Reinforcement Learning with VerifiableRewards (RLVR) for enhancing the reasoning capabilities of large languagemodels (LLMs). Current data selection methods are largely heuristic-based,lacking theoretical guarantees and generalizability. This work proposes atheoretically-grounded approach using influence functions to estimate thecontribution of each data point to the learning objective. To overcome theprohibitive computational cost of policy rollouts required for online influenceestimation, we introduce an off-policy influence estimation method thatefficiently approximates data influence using pre-collected offlinetrajectories. Furthermore, to manage the high-dimensional gradients of LLMs, weemploy sparse random projection to reduce dimensionality and improve storageand computation efficiency. Leveraging these techniques, we develop\textbf{C}urriculum \textbf{R}L with \textbf{O}ff-\textbf{P}olicy\text{I}nfluence guidance (\textbf{CROPI}), a multi-stage RL framework thatiteratively selects the most influential data for the current policy.Experiments on models up to 7B parameters demonstrate that CROPI significantlyaccelerates training. On a 1.5B model, it achieves a 2.66x step-levelacceleration while using only 10\% of the data per stage compared tofull-dataset training. Our results highlight the substantial potential ofinfluence-based data selection for efficient RLVR.