GRPO: Group Relative Policy Optimization DeepSeek's RL Breakthrough

Introduction

DeepSeek-R1 shocked the AI world by achieving GPT-4 level reasoning capabilities through pure reinforcement learning. At the core of this breakthrough is GRPO (Group Relative Policy Optimization), an innovative reinforcement learning algorithm that eliminates the traditional critic network and instead optimizes policy through group-relative rewards.

GRPO solves the core problems of PPO (Proximal Policy Optimization): complexity, instability, and high memory consumption. Through clever group sampling design, GRPO achieves more efficient and stable training, ultimately enabling DeepSeek-R1’s reasoning breakthrough.

To understand why GRPO matters, it helps to recall where reinforcement learning for large language models stood before DeepSeek-R1. The dominant approach was PPO, an actor-critic algorithm that had powered earlier alignment successes but carried a heavy operational burden: a learned value network, multiple supporting models, and a dense web of hyperparameters. Every step of that pipeline consumed GPU memory and required careful tuning, and the value network in particular was a constant source of instability. GRPO’s contribution is to show that, for language-model training, most of that machinery is unnecessary — the reward for a sampled response can be compared against the other responses sampled for the same prompt, and that relative comparison provides all the learning signal a critic would have provided, at a fraction of the cost.

This guide walks through GRPO from first principles: the problems with PPO that motivated it, the loss function at its core, a complete training implementation, and the way DeepSeek-R1 configured it for reasoning. Along the way you will see practical guidance on group sizes, KL schedules, reward design, and the common pitfalls to avoid.

Problems with PPO

Traditional Actor-Critic Architecture

PPO belongs to the Actor-Critic family of reinforcement learning algorithms, which means it needs at least two neural networks working in tandem: an actor that learns the policy, and a critic that estimates the value of each state. The critic’s job is to predict expected future rewards, and the policy update depends on comparing actual outcomes against those predictions. In theory this produces lower-variance gradient estimates, but in practice the value function is hard to learn accurately and it consumes a large fraction of the training budget.

The code below illustrates just how heavy a PPO setup is. Beyond the actor and critic, training typically requires target networks that are updated slowly for stability, and a frozen reference model whose role is to keep the current policy from drifting too far from the original supervised model via a KL penalty. That is four distinct networks in memory at once, and for a 7-billion-parameter language model each one demands gigabytes of GPU memory. The ppo_loss method also shows the full machinery of the clipped surrogate objective and the value loss — every one of those terms is a hyperparameter-laden piece of complexity that GRPO will eventually eliminate.

class PPOArchitecture:
    """
    Traditional PPO requires multiple networks
    """
    
    def __init__(self, state_dim, action_dim):
        # Actor: learns the policy (what to do)
        self.actor = ActorNetwork(state_dim, action_dim)
        
        # Critic: estimates future rewards (value function)
        self.critic = CriticNetwork(state_dim)
        
        # Target networks for stability
        self.target_actor = ActorNetwork(state_dim, action_dim)
        self.target_critic = CriticNetwork(state_dim)
        
        # Reference model for KL constraint
        self.ref_model = ActorNetwork(state_dim, action_dim)
        
    def ppo_loss(self, states, actions, old_log_probs, advantages):
        """
        PPO Clip Objective:
        L(θ) = E[min(r(θ) * A, clip(r(θ), 1-ε, 1+ε) * A)]
        
        Where r(θ) = π_θ(a|s) / π_θ_old(a|s)
        """
        
        # Get current policy probabilities
        new_log_probs = self.actor.get_log_prob(states, actions)
        
        # Compute probability ratio
        ratio = torch.exp(new_log_probs - old_log_probs)
        
        # Clipped surrogate objective
        surr1 = ratio * advantages
        surr2 = torch.clamp(ratio, 1 - 0.2, 1 + 0.2) * advantages
        
        # Take minimum (pessimistic bound)
        policy_loss = -torch.min(surr1, surr2).mean()
        
        # Value function loss
        values = self.critic(states)
        value_loss = F.mse_loss(values, advantages)
        
        return policy_loss + 0.5 * value_loss

The ppo_loss snippet also reveals why PPO is so fiddly to run in production. The probability ratio is clipped to keep updates small, the value loss must be weighted relative to the policy loss, and the advantage estimates themselves come from Generalized Advantage Estimation (GAE), which requires its own discount and lambda parameters to tune. Every new hyperparameter is a place where training can silently diverge, and diagnosing which one went wrong is a slow, expensive process when each experiment costs days of GPU time.

Once you see this machinery, it is easier to appreciate what GRPO removes. Instead of four networks, GRPO keeps the actor and a reference model. Instead of a learned value function, it uses the empirical statistics of a batch of sampled responses. Instead of a half-dozen loss terms, it optimizes a single, self-contained objective. The simplification is not cosmetic — it cuts memory in half, removes a whole class of stability problems, and reduces the number of knobs you must tune.

Four Major Challenges of PPO

The dictionary below condenses these four challenges into the concrete symptoms you will hit when scaling PPO to large language models. The memory figure alone — 40 GB or more for a 7B model — forces teams onto larger GPU clusters or aggressive offloading tricks. The hyperparameter count means every new task or dataset triggers a search over clipping ranges, advantage smoothing, and loss weights. Instability compounds the problem: value networks are notorious for producing exploding gradients, which is why PPO adds gradient clipping and slow target-network updates on top of everything else. And the complexity of GAE means that even a correct implementation is hard to audit, because the advantage estimates feed into the objective in a way that is several layers removed from the raw rewards.

What is striking is that all four problems share a single root cause: the critic. GRPO’s core design decision is to remove the critic entirely and derive advantages from a group of sampled responses instead. Keep that insight in mind as you read the rest of this guide, because every simplification that follows — the reduced memory, the faster convergence, the smaller hyperparameter surface — flows from eliminating that one network.

ppo_problems = {
    'multiple_models': '4 models needed: actor, critic, reference, target',
    'hyperparameters': 'Requires fine tuning: clip epsilon, GAE lambda, value loss weight',
    'instability': 'Gradients may explode, needs gradient clipping and target network updates',
    'memory': '40GB+ GPU memory for 7B model',
    'complexity': 'GAE (Generalized Advantage Estimation) computation is complex',
    
    # Code complexity comparison
    'code_comparison': '''
    PPO requires:
    - advantage = compute_gae(rewards, values, gamma=0.99, lambda=0.95)
    - ratio = (new_policy / old_policy).exp()
    - clipped_ratio = ratio.clamp(1-eps, 1+eps)
    - loss = -min(ratio * advantage, clipped_ratio * advantage)
    - loss += 0.5 * value_loss + 0.01 * entropy_loss
    '''
}

GRPO Core Principles

Key Insight

The core insight of GRPO is: for the same question, we can generate multiple responses and compare their relative quality, rather than learning an absolute value function.

This is a genuinely different philosophy from PPO. PPO asks “how good is this state or action in absolute terms?” and needs a critic trained on billions of value predictions to answer. GRPO asks a much simpler question: “how good is this response compared to the other responses we sampled for the same prompt?” Comparing within a group requires no learned baseline at all — just the rewards of the responses themselves. Since most language-model reward functions (correctness checks, format compliance, ground-truth matching) are computable directly, the relative ranking is available for free.

The sketch below captures the essence. For each prompt, GRPO samples a group of G responses from the current policy, scores each with the reward function, and then normalizes those scores using the group’s own mean and standard deviation. A response that scores above the group average gets a positive advantage and is reinforced; one below the average is suppressed. This zero-sum, within-group comparison is exactly what removes the need for a value network, because the group statistics play the role that the critic used to play — and they are exact rather than learned.

def grpo_key_insight():
    """
    GRPO key insight:
    
    For each prompt q, we sample G responses {o_1, o_2, ..., o_G}
    from the old policy π_ref
    
    Then compute each response's reward r(o_i)
    
    Use within-group statistics as baseline:
    - mean: group average reward
    - std: group reward standard deviation
    
    Advantage function: A_i = (r(o_i) - mean) / std
    
    This eliminates the need for a value network!
    """
    pass

This zero-sum property has a practical consequence worth internalizing: GRPO cares about the relative ordering of responses, not their absolute reward values. That makes it robust to reward scaling — if you multiply every reward by a constant, the normalized advantages are unchanged, so the gradient direction is identical. In practice this means teams can combine several reward terms with simple additive weights and still get stable training, which is exactly the flexibility DeepSeek-R1 exploited with its composite rewards.

GRPO Loss Function

The grpo_loss function below is the mathematical heart of GRPO, and every line maps to the insight just described. It takes the logits from the policy model and the frozen reference model for a batch of grouped responses, along with the raw rewards, and produces a single scalar loss to minimize.

Working through the code in order: it first converts both sets of logits into log-probabilities, then sums over the token dimension so that each response becomes a single scalar log-probability. Next, it computes the group mean and standard deviation of the rewards, adding a tiny epsilon to the standard deviation to avoid division by zero, and normalizes to get advantages. The policy update term is the difference between the current policy’s log-probability and the reference’s, scaled by the advantage — which naturally makes the better-than-average responses more likely. Finally, the KL penalty is added as a separate term, weighted by beta, keeping the policy close to the reference model.

import torch
import torch.nn.functional as F

def grpo_loss(
    policy_logits,     # policy model logits: [batch, group_size, seq_len, vocab]
    ref_logits,        # reference model logits
    rewards,           # reward values: [batch, group_size]
    beta: float = 0.1,
    epsilon: float = 0.2
):
    """
    GRPO loss function
    
    Args:
        policy_logits: policy model output
        ref_logits: reference (SFT) model output
        rewards: reward for each response [batch, group_size]
        beta: KL penalty coefficient
        epsilon: clipping parameter
        
    Returns:
        loss: GRPO loss value
    """
    batch_size, group_size, seq_len, vocab_size = policy_logits.shape
    
    # Compute log probabilities
    policy_logprobs = F.log_softmax(policy_logits, dim=-1)
    ref_logprobs = F.log_softmax(ref_logits, dim=-1)
    
    # Get total log probability per response (sum over tokens)
    # Requires attention mask to ignore padding
    log_probs = policy_logprobs.sum(dim=(2, 3))  # [batch, group_size]
    ref_log_probs = ref_logprobs.sum(dim=(2, 3))  # [batch, group_size]
    
    # Compute group-relative rewards (advantages)
    # Computed for all responses of each prompt
    mean_reward = rewards.mean(dim=1, keepdim=True)  # [batch, 1]
    std_reward = rewards.std(dim=1, keepdim=True) + 1e-8  # [batch, 1]
    
    advantages = (rewards - mean_reward) / std_reward  # [batch, group_size]
    
    # Compute policy gradient term
    # log π(a_i | q) - log π_ref(a_i | q)
    policy_ref_diff = log_probs - ref_log_probs  # [batch, group_size]
    
    # Weighted advantage
    weighted_diff = policy_ref_diff * advantages  # [batch, group_size]
    
    # Add KL penalty term
    kl_penalty = (ref_log_probs - log_probs)  # [batch, group_size]
    
    # Final loss: maximize advantage + KL regularization
    loss = -(weighted_diff - beta * kl_penalty).mean()
    
    return loss

Two design details in this objective are easy to overlook but worth flagging. The KL penalty is computed as ref_log_probs - log_probs, so it grows when the current policy diverges from the reference, and the negative sign in front of the whole expression means we are maximizing the advantage while minimizing that divergence. Note that GRPO’s version here applies the KL term per-response and weighs it by beta, which is simpler than PPO’s per-token KL controller yet achieves the same protective effect in practice. Also observe that there is no clipping term in this formulation; GRPO relies on the KL constraint to keep updates bounded, whereas PPO relied on a clip range.

Because the loss is purely a function of log-probabilities and rewards, it has no tunable advantage estimator and no value head. That is the entire point: the objective is small, interpretable, and — as the next section shows — trivial to wrap into a complete training loop.

Complete GRPO Implementation

The GRPOTrainer class below turns the loss into a working training loop, and it is useful to trace the data flow because it shows how the group dimension threads through every stage. The constructor takes the policy model to be trained, a frozen reference model, a reward function, and the two key hyperparameters: beta for the KL penalty and group_size for how many responses are sampled per prompt.

sample_responses is where the group structure is born: for every prompt, the trainer calls generate group_size times with sampling enabled, so the same prompt yields several distinct responses. compute_rewards then scores every response in every group, returning a tensor shaped [batch, group_size]. The forward_batch method flattens the groups for a single batched forward pass through both the policy and the reference model, reshapes the logits back to the [batch, group_size, seq_len, vocab] shape the loss expects, and combines everything. This batching is important: running one forward pass over all groups is far cheaper than launching group_size separate passes.

classGRPOTrainer:
    """
    Complete GRPO training implementation
    """
    
    def __init__(
        self,
        policy_model,      # policy model to train
        ref_model,        # reference model (frozen SFT model)
        reward_fn,        # reward function
        beta: float = 0.1,
        group_size: int = 4,
        max_length: int = 512
    ):
        self.policy_model = policy_model
        self.ref_model = ref_model
        self.reward_fn = reward_fn
        self.beta = beta
        self.group_size = group_size
        self.max_length = max_length
        
        # Freeze reference model
        for param in ref_model.parameters():
            param.requires_grad = False
    
    def sample_responses(self, prompts):
        """
        Sample multiple responses for each prompt
        """
        all_responses = []
        
        for prompt in prompts:
            # Sample multiple times to generate multiple responses
            responses = []
            for _ in range(self.group_size):
                response = self.policy_model.generate(
                    prompt,
                    max_new_tokens=self.max_length,
                    do_sample=True,
                    temperature=0.7,
                )
                responses.append(response)
            
            all_responses.append(responses)
        
        return all_responses
    
    def compute_rewards(self, prompts, responses):
        """
        Compute reward for each response
        """
        all_rewards = []
        
        for prompt, response_group in zip(prompts, responses):
            # Compute reward for each response in the group
            group_rewards = []
            for response in response_group:
                reward = self.reward_fn(prompt, response)
                group_rewards.append(reward)
            
            all_rewards.append(group_rewards)
        
        return torch.tensor(all_rewards, dtype=torch.float32)
    
    def forward_batch(self, prompts, responses):
        """
        Forward pass to compute loss
        """
        batch_size = len(prompts)
        
        # Prepare data
        # [batch * group_size, seq_len]
        flattened_responses = [r for group in responses for r in group]
        
        # Tokenize
        inputs = self.tokenizer(
            flattened_responses,
            return_tensors='pt',
            padding=True,
            truncation=True,
            max_length=self.max_length
        )
        
        # Policy model forward
        policy_outputs = self.policy_model(
            input_ids=inputs['input_ids'],
            attention_mask=inputs['attention_mask']
        )
        
        # Reference model forward (no gradient)
        with torch.no_grad():
            ref_outputs = self.ref_model(
                input_ids=inputs['input_ids'],
                attention_mask=inputs['attention_mask']
            )
        
        # Reshape to [batch, group_size, seq_len, vocab]
        policy_logits = policy_outputs.logits.view(
            batch_size, self.group_size, -1, self.policy_model.config.vocab_size
        )
        ref_logits = ref_outputs.logits.view(
            batch_size, self.group_size, -1, self.ref_model.config.vocab_size
        )
        
        # Compute rewards
        rewards = self.compute_rewards(prompts, responses)
        
        # Compute GRPO loss
        loss = grpo_loss(
            policy_logits,
            ref_logits,
            rewards,
            beta=self.beta
        )
        
        return loss
    
    def train_step(self, prompts):
        """
        Single training step
        """
        # 1. Sample responses
        responses = self.sample_responses(prompts)
        
        # 2. Forward pass and loss computation
        loss = self.forward_batch(prompts, responses)
        
        # 3. Backward pass
        self.optimizer.zero_grad()
        loss.backward()
        
        # Gradient clipping
        torch.nn.utils.clip_grad_norm_(self.policy_model.parameters(), 1.0)
        
        self.optimizer.step()
        
        return loss.item()

The train_step method reveals the complete rhythm of a GRPO iteration: sample, score, forward, backprop. Because the reference model is frozen and only the policy receives gradients, the memory footprint stays near half of PPO’s — there is no critic and no target network to hold in memory. Note also that the trainer never constructs an explicit advantage buffer or value head; the advantages materialize inside the loss computation and die on the backward pass. For teams running at DeepSeek’s scale, those savings translate directly into faster iteration and the ability to fit larger batches on the same hardware, which is precisely why GRPO made the DeepSeek-R1 reasoning project tractable.

DeepSeek-R1 Application

GRPO’s Role in DeepSeek-R1

DeepSeek-R1 is the flagship demonstration of GRPO, and its training configuration shows how the algorithm’s knobs are set in practice. The DeepSeekR1Training class below wires together the pieces introduced earlier: a base policy model, a supervised fine-tuned reference model, and a composite reward function.

The most instructive parts are the configuration choices in train. DeepSeek used a much larger group size — sixteen responses per prompt rather than the four or eight you might start with — because the bigger the group, the more reliable the mean-and-standard-deviation baseline becomes for difficult reasoning problems. They also used a smaller beta of 0.04, allowing the policy more room to explore while still staying anchored to the reference model. And the training ran for many thousands of steps over the same prompts, which is the recipe that turned a language model that could not do arithmetic into one that produces long, structured reasoning chains.

class DeepSeekR1Training:
    """
    DeepSeek-R1 uses GRPO for reasoning capability training
    """
    
    def __init__(self):
        self.base_model = None
        self.reward_functions = []
    
    def setup_rewards(self):
        """
        R1 uses a combination of multiple reward functions
        """
        # 1. Accuracy reward: check if answer is correct
        self.reward_functions.append(AccuracyReward())
        
        # 2. Format reward: require model to use specific format
        self.reward_functions.append(FormatReward())
        
        # 3. Reasoning step reward: check thought process
        self.reward_functions.append(ReasoningReward())
    
    def compute_composite_reward(self, prompt, response):
        """
        Combine multiple rewards
        """
        total_reward = 0.0
        
        for reward_fn in self.reward_functions:
            reward = reward_fn(prompt, response)
            total_reward += reward
        
        return total_reward
    
    def train(self, prompts):
        """
        Train using GRPO
        """
        trainer = GRPOTrainer(
            policy_model=self.base_model,
            ref_model=self.sft_model,
            reward_fn=self.compute_composite_reward,
            group_size=16,  # DeepSeek uses larger group
            beta=0.04       # Smaller beta
        )
        
        for step in range(10000):
            loss = trainer.train_step(prompts)
            
            if step % 100 == 0:
                print(f"Step {step}: Loss = {loss:.4f}")

The composite-reward design deserves a closer look, because it is how DeepSeek encoded the “right” behavior without a learned reward model. Rather than training a reward model from human preferences — which PPO-style pipelines usually need — R1 combined three hand-crafted, rule-based signals: whether the final answer is correct, whether the response follows the required <think>/<answer> format, and whether the reasoning chain has substance. Each signal carries a weight, and the sum drives the group-relative comparison. Rule-based rewards are exactly the setting where GRPO shines, because they are cheap, deterministic, and unambiguous to compare within a group.

Reward Function Design

The reward functions below show how each signal is implemented. AccuracyReward is the strict one: it extracts the final answer, compares it against a ground truth, and returns its full weight only on an exact match. This is the reward that does the heavy lifting for reasoning capability, and it is why GRPO is so effective on math and code — the correctness signal is unambiguous.

FormatReward is more forgiving, returning the full weight when the response contains both <think> and <answer> tags and half weight when only one is present. It exists to teach the model a structured output style, and its small weight reflects the fact that formatting is secondary to correctness. ReasoningReward completes the trio by rewarding longer thinking blocks, normalized so that very long chains do not dominate. Together the three rewards illustrate a general design principle for GRPO: keep each signal simple and rule-based, weight them so that correctness dominates, and let the group comparison resolve the rest.

class AccuracyReward:
    """
    Accuracy reward: checks if final answer is correct
    """
    
    def __init__(self):
        self.weight = 1.0
    
    def __call__(self, prompt, response):
        # Extract answer and check correctness
        extracted_answer = self.extract_answer(response)
        ground_truth = self.get_ground_truth(prompt)
        
        if extracted_answer == ground_truth:
            return self.weight
        else:
            return 0.0
    
    def extract_answer(self, response):
        # Extract answer from model response
        # May need regex or special markers
        pass
    
    def get_ground_truth(self, prompt):
        # Get correct answer from the question
        pass


class FormatReward:
    """
    Format reward: requires model output to include thought process
    """
    
    def __init__(self):
        self.weight = 0.1
    
    def __call__(self, prompt, response):
        # Check if response contains <think> tags
        has_think = '<think>' in response and '</think>' in response
        has_answer = '<answer>' in response and '</answer>' in response
        
        if has_think and has_answer:
            return self.weight
        elif has_think or has_answer:
            return self.weight * 0.5
        else:
            return 0.0


class ReasoningReward:
    """
    Reasoning reward: encourages long reasoning chains
    """
    
    def __init__(self):
        self.weight = 0.01
    
    def __call__(self, prompt, response):
        # Reward longer reasoning processes
        # But only when format is correct
        think_content = self.extract_think(response)
        reasoning_length = len(think_content)
        
        # Normalize: longer length gets higher reward (with upper bound)
        normalized_reward = min(reasoning_length / 1000, 1.0)
        
        return self.weight * normalized_reward

Performance Analysis

GRPO vs PPO

The most persuasive argument for GRPO is not theoretical elegance but measured results, and the comparison below lays out the numbers in the dimensions that matter most to practitioners: memory, speed, efficiency, stability, and hyperparameter burden.

The headline figures are the memory and speed improvements. Halving memory from 40 GB to 20 GB for a 7B model is not just a cost saving — it determines whether a training run fits on a single GPU at all, and it changes which hardware configurations are feasible for smaller teams. The 2x training-speed gain compounds over thousands of steps, turning a multi-day experiment into an overnight one and enabling the rapid iteration that reasoning-model development demands. Just as importantly, GRPO replaces PPO’s learned, potentially biased value estimates with an empirical baseline computed directly from the sampled group, which removes a systematic source of error.

# Performance comparison
performance_comparison = {
    'memory_usage': {
        'PPO': '40GB+ for 7B model',
        'GRPO': '20GB for 7B model',  # 50% reduction
    },
    'training_speed': {
        'PPO': '3 days on 8x A100',
        'GRPO': '1.5 days on 8x A100',  # 2x faster
    },
    'sample_efficiency': {
        'PPO': 'Uses value estimation, can be biased',
        'GRPO': 'Empirical baseline, more accurate',
    },
    'stability': {
        'PPO': 'Requires clipping, value loss weighting',
        'GRPO': 'Simple objective, more stable',
    },
    'hyperparameters': {
        'PPO': '10+ hyperparameters',
        'GRPO': '2-3 key hyperparameters (beta, group_size)',
    }
}

The stability and hyperparameter rows deserve special attention because they capture the operational cost of each algorithm. PPO’s ten-plus hyperparameters — clip epsilon, GAE lambda, value-loss weight, entropy bonus, gradient thresholds — each need to be re-tuned whenever the task or data changes, and a bad choice can silently destabilize training for hours. GRPO reduces the surface to two or three knobs: the group size and the KL weight, with the clipping range optionally playing a supporting role. Teams that have run both consistently report that GRPO’s simplicity is not just pleasant — it is what makes large-scale reasoning training manageable at all.

Math Reasoning Results

The math benchmarks below tell the story that made DeepSeek-R1 famous. On GSM8K, a widely used grade-school math benchmark, GRPO-trained models more than double the performance of PPO-trained ones and leap from 15.6 percent on the base model to 89.3 percent. On the harder MATH benchmark the gap narrows but still favors GRPO substantially. These are not marginal gains; they represent a qualitative shift in capability, from models that occasionally stumble onto the right answer to models that can consistently reason through multi-step problems.

It is worth noting why the group-relative signal helps so much on these tasks. On math problems, correctness is binary and unambiguous, which means the rewards within a group are easy to compare and the advantages are highly informative — the model gets a clear, strong signal about which sampled responses were right. This is precisely the condition under which GRPO’s empirical baseline is most reliable. The dramatic results are thus not magic but the natural payoff of matching the algorithm to a problem where the reward function is clean and objective.

# DeepSeek-R1 performance on math reasoning tasks
math_results = {
    'GSM8K': {
        'base_model': '15.6%',
        'PPO_trained': '52.3%',
        'GRPO_trained': '89.3%',  # Significantly higher
    },
    'MATH': {
        'base_model': '10.2%',
        'PPO_trained': '28.5%',
        'GRPO_trained': '47.1%',
    }
}

Implementation Details

Group Size Selection

Choosing group_size is the single most consequential decision in a GRPO setup, and it embodies a direct trade-off. A larger group gives a more accurate estimate of the mean and standard deviation, which makes the advantages less noisy and training more stable. But every extra response per prompt costs a full generation pass through the model, and — because the loss is averaged over groups — a larger group also means fewer distinct prompts get a gradient update per batch. You are trading baseline fidelity against throughput and coverage.

The heuristic function below turns that trade-off into a concrete rule based on task complexity. Simple tasks with mostly-correct responses can get away with small groups, because the group statistics are reliable even with four samples. Hard reasoning tasks, where most sampled responses fail, need larger groups so that the rare correct answers still stand out against the average. DeepSeek-R1 settled on sixteen for exactly this reason: on difficult math problems, a small group of mostly-wrong responses produces a noisy, low-information baseline, while a larger group gives the model a clearer sense of what success looks like relative to failure.

def optimal_group_size(task_complexity):
    """
    Select group size based on task complexity
    
    Args:
        task_complexity: Task complexity score 1-10
        
    Returns:
        optimal_group_size
    """
    if task_complexity <= 3:
        # Simple tasks, smaller group works
        return 4
    elif task_complexity <= 6:
        # Medium complexity
        return 8
    else:
        # High complexity reasoning tasks
        return 16  # DeepSeek-R1 uses 16
    
    # General rules:
    # - Larger groups provide more accurate baseline estimation
    # - But increasing group size reduces gradient updates per epoch
    # - In practice, 4-16 is the common range

Beta Scheduling

The KL weight beta controls how far the policy may drift from the reference model, and its value changes the character of training. A high beta keeps the model conservative and close to its supervised starting point; a low beta lets it explore more freely but risks the output degrading into repetition or gibberish. In practice the optimal policy is not a constant — early in training you want the safety of a high penalty, while later you want to relax it so the model can commit to the strong reasoning behaviors the rewards are teaching it.

The cosine_beta_schedule function below encodes exactly that progression. During the first ten percent of training, beta stays at its high starting value, holding the policy near the reference while the reward signal accumulates. In the middle stretch, a cosine curve smoothly anneals beta down to its target. In the final phase, beta holds at the low ending value, giving the model maximum freedom to finalize its learned behavior. This kind of schedule — fast off the start, smooth in the middle, flat at the end — is a widely reusable pattern for any GRPO run, and the two numbers you must pick (start_beta and end_beta) are far easier to reason about than PPO’s full hyperparameter matrix.

def cosine_beta_schedule(total_steps, start_beta=0.1, end_beta=0.04):
    """
    Beta scheduling: gradually reduce KL penalty to allow larger policy changes
    """
    def get_beta(step):
        if step < total_steps * 0.1:
            # Early stage: high beta, stay close to reference model
            return start_beta
        elif step > total_steps * 0.8:
            # Late stage: low beta, allow more exploration
            return end_beta
        else:
            # Middle stage: cosine annealing
            progress = (step - total_steps * 0.1) / (total_steps * 0.7)
            return start_beta - (start_beta - end_beta) * (1 + torch.cos(torch.tensor(progress * torch.pi))) / 2
    
    return get_beta

Advanced Variant: GRPO with Self-Consistency

GRPO is a framework, not a fixed recipe, and one of the most effective variations combines it with self-consistency — a technique that samples many responses and picks the answer that appears most often. Self-consistency exploits the observation that for deterministic problems, multiple independent reasoning attempts tend to converge on the correct answer even when a single attempt is unreliable.

The implementation below merges the two ideas. It first generates a large pool of responses in groups, then uses majority voting across each group to identify the consensus answer. Responses that agree with the consensus receive a high reward, while dissenting ones receive a small negative reward. Feeding those rewards back through the standard GRPO loss has a powerful effect: instead of simply rewarding any correct-looking response, the model learns to produce the kind of reasoning that agrees with the majority — which, on math and code problems, correlates strongly with correctness. This variant typically needs a smaller group than plain GRPO, because the consensus signal already provides much of the variance reduction that a large group would.

def grpo_with_self_consistency(
    policy_model,
    ref_model,
    prompts,
    group_size=8,
    num_final_samples=16
):
    """
    GRPO combined with self-consistency
    1. Generate multiple responses
    2. Use majority voting to select the most consistent answer
    3. Give higher reward to responses matching the majority
    """
    
    all_responses = []
    for _ in range(group_size):
        responses = policy_model.generate(prompts)
        all_responses.append(responses)
    
    # Extract all answers
    all_answers = [[extract_answer(r) for r in group] for group in all_responses]
    
    # Majority voting
    final_answers = []
    for answer_group in all_answers:
        # Count occurrences of each answer
        from collections import Counter
        counts = Counter(answer_group)
        # Most common answer is the final answer
        final_answer = counts.most_common(1)[0][0]
        final_answers.append(final_answer)
    
    # Reward: higher reward for responses matching the final consistent answer
    rewards = []
    for group_answers in all_answers:
        group_rewards = []
        majority_count = max(Counter(group_answers).values())
        for answer in group_answers:
            if answer == final_answers[0]:
                reward = 1.0
            else:
                reward = -0.1
            group_rewards.append(reward)
        rewards.append(group_rewards)
    
    # Use standard GRPO loss
    return grpo_loss(policy_logits, ref_logits, rewards)

One caveat about the self-consistency variant is worth stating plainly: it multiplies inference cost. Every additional sampled response is a full generation through the model, and because the reward now depends on the whole group’s consensus rather than per-response ground truth, you cannot prune the pool early. That makes it most attractive in offline or batch settings where the sampled responses are reused across many training steps, amortizing the generation cost. For online reinforcement learning with tight latency budgets, plain GRPO with a well-chosen group size remains the more economical choice.

Comparison with DPO

The dictionary below lays out the key differences between GRPO and DPO side by side, so you can see at a glance where the two algorithms agree and where they diverge in their training signals, sampling requirements, and target use cases.

# GRPO vs DPO comparison
comparison = {
    'training_signal': {
        'DPO': 'Pairwise preference: chosen vs rejected',
        'GRPO': 'Group relative rewards: relative ranking of multiple responses',
    },
    'reference_model': {
        'DPO': 'Required (computes KL)',
        'GRPO': 'Required (computes KL)',
    },
    'sampling': {
        'DPO': '2 responses per prompt',
        'GRPO': 'G responses per prompt (G >= 4)',
    },
    'reward_type': {
        'DPO': 'Binary preference',
        'GRPO': 'Continuous reward',
    },
    'use_case': {
        'DPO': 'General preference alignment',
        'GRPO': 'Reasoning capability enhancement',
    }
}

The comparison above also clarifies when each algorithm should be your first choice. DPO is a direct-preference algorithm: it needs pairs of chosen and rejected responses and optimizes the model to rank the chosen one higher, with no sampling loop and no rewards at inference time. That makes it lightweight and ideal for aligning a model to human preferences, where the signal is inherently pairwise. GRPO, by contrast, needs a continuous reward and a sampling loop, but it can exploit arbitrary scalar signals — including the deterministic correctness checks that are so powerful for math and code. If your objective is measurable and you can afford the generation cost, GRPO’s richer signal wins; if you only have binary preference judgments, DPO’s simplicity is the right trade.

Practical Advice

When to Use GRPO

Knowing where an algorithm shines is half the battle; knowing where it fails is the other half, and GRPO’s strengths and weaknesses both trace back to the group-relative baseline. The list below summarizes the ideal conditions: tasks with clear, rule-based correctness criteria, where a model can often verify its own output, where memory is constrained, and where fast iteration matters. Math, code, and logical reasoning problems check every box.

The exclusion list is equally instructive. GRPO struggles with subjective preferences, where there is no crisp ground truth to compare within a group — two human raters may disagree on the “better” response, so the group statistics become arbitrary. It also fares poorly in complex interactive environments, where the reward depends on long-horizon interactions and the immediate group comparison loses meaning. For those cases you likely need a learned reward model (which reintroduces the complexity GRPO removes) or a DPO-style preference objective. Matching the algorithm to the reward structure is the difference between a breakthrough and a frustrating training run.

# GRPO ideal use cases
grpo_ideal_cases = {
    'reasoning_tasks': 'Math, code, logical reasoning',
    'self_verification': 'Model can verify its own output',
    'rule_based_rewards': 'Clear correctness criteria available',
    'limited_memory': 'Cannot afford PPO memory overhead',
    'quick_iteration': 'Need fast experimentation and iteration',
    
    # Not suitable:
    'subjective_preferences': 'Subjective preferences have no clear standard',
    'complex_environments': 'Need to interact with complex environments',
}

Common Pitfalls

Even a beautifully simple objective can go wrong in practice, and the issues below are the ones that recur most often in GRPO training runs. High reward variance is the most common: if the rewards within a group are all over the map, the normalization can amplify noise rather than signal, which is why enlarging the group or normalizing the rewards before comparison is the standard first fix.

The remaining pitfalls are symptoms of the interaction between the reward design and the KL constraint. A model that starts repeating the same response over and over is a sign that the reward landscape rewards verbosity or repetition, and a dedicated repetition-penalty term is the targeted remedy. A KL divergence that explodes indicates the policy is outrunning its anchor, which means raising beta. And instability in general — loss spikes, sudden quality drops — is the training telling you the gradient magnitude is out of control, so gradient clipping and a lower learning rate are the right levers. Keeping this checklist in mind turns most debugging sessions into a five-minute diagnosis rather than a week of experiments.

# GRPO common issues and solutions
common_issues = {
    'issue1': {
        'problem': 'Reward variance too high',
        'solution': 'Increase group size or use reward normalization'
    },
    'issue2': {
        'problem': 'Model starts repeating responses',
        'solution': 'Add repetition penalty reward term'
    },
    'issue3': {
        'problem': 'KL divergence too large',
        'solution': 'Increase beta value'
    },
    'issue4': {
        'problem': 'Unstable training',
        'solution': 'Use gradient clipping, reduce learning rate'
    }
}

Conclusion

GRPO represents a major breakthrough in reinforcement learning optimization:

• Memory halved: 50% memory usage reduction
• Speed doubled: 2x faster training speed
• More stable: Fewer hyperparameters, more reliable convergence
• Reasoning breakthrough: Enabled DeepSeek-R1’s math reasoning capabilities

By replacing value networks with group-relative rewards, GRPO greatly simplifies the reinforcement learning pipeline while maintaining — and even improving — training effectiveness. This algorithm is becoming the new standard for reasoning model training.

For practitioners, the takeaway is a decision framework. Start with a clear, rule-based reward signal and a modest group size, then scale the group up for hard reasoning tasks and anneal the KL penalty over training. If the reward is binary preference data rather than a continuous correctness signal, prefer DPO. If you need interactive, long-horizon control, you are outside GRPO’s sweet spot entirely. Within its sweet spot — math, code, and other verifiable reasoning — GRPO delivers the kind of memory savings, training speed, and stability that make large-scale reinforcement learning practical, and its simplicity means you spend your time improving the reward and the data rather than fighting the algorithm.

Resources

• GRPO Paper: DeepSeek-R1 Technical Report
• DeepSeek-R1 GitHub
• GRPO Official Implementation
• HuggingFace GRPO Tutorial