Group Relative Policy Optimization

A groupwise alignment training regime that samples multiple candidate answers for the same prompt and updates the model from how those answers rank relative to one another.

Group Relative Policy Optimization, usually shortened to GRPO, is a post-training alignment method that samples several candidate answers for the same prompt, scores them as a group, and nudges the model toward answers that rank higher relative to the other samples in that group.

At a glance

Released

February 2024

Authors

Zhihong Shao, Peiyi Wang, Qihao Zhu, et al.

Regime type

Training Alignment

Related modules

No related modules listed yet.

What It Is

Group Relative Policy Optimization is a post-training alignment regime for language models. For each prompt, the current model samples a small group of candidate answers, compares them against one another, and uses that within-group ranking as the training signal instead of learning from one fixed preferred-versus-rejected pair.

Why It Exists

During post-training, teams often need updates that react to freshly sampled model behavior instead of only to comparisons frozen in a dataset. A single prompt can produce several candidate answers with different strengths, and the useful signal is often how those answers rank relative to one another in the same group. Group Relative Policy Optimization exists so post-training can learn from those grouped relative rankings rather than from one static preference pair at a time.

How It Works

For each training prompt, the current model samples several candidate answers from the same policy. A reward or preference signal scores every answer in that group. Group Relative Policy Optimization then compares those scores relative to one another: answers that beat the group average receive a positive training signal, while weaker answers are pushed down. The policy update follows that within-group ranking instead of a single fixed preferred-versus-rejected pair.
Group Relative Policy Optimization training flow
Request and weight flow
GRPO compares sampled answers within each prompt group before applying the policy update.
Aiapproxfracri−barrGsigmaG+epsilonA_i \\approx \\frac{r_i - \\bar{r}_G}{\\sigma_G + \\epsilon}

Ai

relative advantage for sample i

ri

reward or preference score for sample i
rG
mean score across the prompt group

σG

standard deviation of scores in the group
ε
small stabilizer constant

Compared To Nearby Regimes

Reinforcement learning from human feedback, often shortened to RLHF, is the broader feedback-driven post-training workflow: collect human rankings or comparisons, shape a reward signal, and update the policy so the model behaves more like what people prefer. Group Relative Policy Optimization is not that whole workflow. It is one specific optimization approach used inside preference-shaped or reward-shaped post-training when teams want grouped relative comparisons over freshly sampled answers. Compared with Proximal Policy Optimization, or PPO, GRPO keeps the policy-learning idea but changes how the update signal is formed. PPO-style loops often maintain a separate value model to estimate how good a trajectory is while the policy explores. GRPO instead samples a small group of answers for the same prompt, scores them together, and updates from how each answer ranks relative to the others in that group—without needing a separate value model for that grouped signal. Compared with Direct Preference Optimization, or DPO, GRPO learns from groups of sampled candidates for one prompt rather than only from stored chosen-versus-rejected pairs. DPO nudges the model toward a fixed preferred answer and away from a fixed rejected one already in the dataset. GRPO lets the current policy produce several live candidates, compares them within the group, and trains on that within-group ranking.

Limitations And Failure Modes

GRPO depends on the quality of the reward or preference signal applied to each sampled group. If scoring is noisy, too sparse, or gameable, the grouped update can reinforce shallow patterns. Sampling several answers per prompt also adds rollout cost relative to methods that train directly from fixed preference datasets.

Tags

References

  1. Shao, Zhihong, et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv, 2024, https://arxiv.org/abs/2402.03300.

Direct Preference Optimization

Previous Page

Instruction Tuning

Next Page

On this page

What It IsWhy It ExistsHow It WorksCompared To Nearby RegimesLimitations And Failure ModesRelated ToTagsReferences