Abstract
Reinforcement Learning (RL) can directly enhance the reasoning capabilitiesof large language models without extensive reliance on Supervised Fine-Tuning(SFT). In this work, we revisit the traditional Policy Gradient (PG) mechanismand propose a minimalist RL approach termed Group Policy Gradient (GPG). Unlikeconventional methods, GPG directly optimize the original RL objective, thusobviating the need for surrogate loss functions. By eliminating the critic andreference models, avoiding KL divergence constraints, and addressing theadvantage and gradient estimation bias, our approach significantly simplifiesthe training process compared to Group Relative Policy Optimization (GRPO). Ourapproach achieves superior performance without relying on auxiliary techniquesor adjustments. As illustrated in Figure 1, extensive experiments demonstratethat our method not only reduces computational costs but also consistentlyoutperforms GRPO across various unimodal and multimodal tasks. Our code isavailable at https://github.com/AMAP-ML/GPG.