Abstract
Reinforcement Learning (RL) can directly enhance the reasoning capabilitiesof large language models without extensive reliance on Supervised Fine-Tuning(SFT). In this work, we revisit the traditional Policy Gradient (PG) mechanismand propose a minimalist RL approach termed Group Policy Gradient (GPG). Unlikeconventional methods, GPG directly optimize the original RL objective, thusobviating the need for surrogate loss functions. As illustrated in our paper,by eliminating both the critic and reference models, and avoiding KL divergenceconstraints, our approach significantly simplifies the training process whencompared to Group Relative Policy Optimization (GRPO). Our approach achievessuperior performance without relying on auxiliary techniques or adjustments.Extensive experiments demonstrate that our method not only reducescomputational costs but also consistently outperforms GRPO across variousunimodal and multimodal tasks. Our code is available athttps://github.com/AMAP-ML/GPG.