Group Relative Policy Optimization (GRPO)

Group-relative policy optimization, known as GRPO, is a widely adopted RL algorithm introduced by DeepSeek in their DeepSeekMath paper in 2024.

GRPO generates multiple rollouts for the same prompt and calculates a reward for each one (with rubrics, LLM as a judge, or any verifiable metric). It finds the mean reward across the group, then calculates each rollout’s advantage by subtracting the mean from its reward. We often divide by the group’s standard deviation to scale the advantages relative to the spread of rewards in the group.

$$A_i=\frac{r_i-\mathrm{mean}(r_1,\dots ,r_G)}{\mathrm{std}(r_1,\dots ,r_G)}$$

Here, $r_i$ is the reward for rollout $i$ and $G$ is the number of rollouts in the group. Essentially, this highlights which rollouts received more reward than the others, and these advantages then inform the trainer to increase the probability of the tokens in the above-average rollouts (and decrease the probability of those in below-average rollouts).

As a result, GRPO is most useful when rollouts for the same prompt receive different rewards. If all rollouts receive the same reward, they provide no relative advantages (especially if they all succeed or all fail) and therefore provide no learning signal.