---
title: Compression Group Relative Policy Optimization
url: https://www.emergentmind.com/topics/compression-group-relative-policy-optimization-c-grpo
type: topic
---

# Compression Group Relative Policy Optimization

Searching arXiv for recent GRPO variants relevant to compression-oriented formulations and C-GRPO terminology.
Compression Group Relative Policy Optimization (C-GRPO) is an umbrella characterization for a family of Group Relative Policy Optimization (GRPO) variants that compress, aggregate, or selectively reduce the optimization payload of group-based reinforcement learning while preserving the defining GRPO principle of within-group relative credit assignment. In the literature, “C-GRPO” is not a standardized method name. Instead, closely related work appears under formulations such as Fine-grained Group Policy Optimization (FGO), Binary Prefix Policy Optimization (BPPO), Soft Sequence Policy Optimization (SSPO), Token-Level Policy Optimization (TEPO), and CoDistill-GRPO, each of which modifies GRPO by compressing reward structure, update sets, token-level corrections, or rollout sources in distinct ways [2602.10048], [2605.28028], [2602.19327], [2510.09369], [2605.08873]. The shared theme is that standard GRPO often propagates coarse sequence-level or group-level signals through token-level updates in a way that is costly, redundant, or statistically brittle; C-GRPO-style methods seek more compact and better-conditioned surrogates without abandoning group-relative optimization.

## 1. Conceptual scope and naming

The expression “Compression Group Relative Policy Optimization” does not currently denote a single canonical algorithm in the arXiv literature. Several papers are directly relevant to such a label, but they adopt different names and emphasize different forms of compression. Some compress long chain-of-thought (CoT) behavior by reward shaping toward shorter valid trajectories, as in FGO [2602.10048]. Others compress the update-bearing subset of completions and tokens, as in BPPO, which keeps only the shortest correct completion and the shortest incorrect completion while preserving full-group advantage normalization [2605.28028]. Still others compress token-level importance information into a sequence-level statistic through geometric aggregation, as in SSPO and TEPO [2602.19327], [2510.09369].

This suggests that “C-GRPO” is best treated as a higher-level editorial category rather than a fixed paper title. Under that interpretation, C-GRPO denotes GRPO-family objectives in which compression occurs along at least one of four axes. First, there is **response compression**, where the policy is explicitly trained to prefer shorter reasoning traces. Second, there is **update-set compression**, where only a subset of sampled completions is used for policy updates. Third, there is **token-signal compression**, where tokenwise policy ratios or gradients are aggregated into a more compact sequence-level form. Fourth, there is **systems-level compression**, where rollout generation or training structure is shifted toward cheaper models or smaller effective batches [2602.10048], [2605.28028], [2602.19327], [2605.08873].

A common misconception is that compression-oriented GRPO necessarily means post hoc truncation of reasoning. The available work does not support that reduction. The central design pattern is instead to alter the RL objective or reward construction so that concise, stable, or non-redundant behavior becomes favorable during training itself [2602.10048], [2605.28028].

## 2. GRPO substrate and the compression problem

Standard GRPO samples a group of responses for the same prompt, computes scalar rewards, normalizes these rewards within the group, and then applies a PPO-style clipped policy update using token-level importance ratios. In one representative formulation, the GRPO objective is
\[
\mathcal{J}_\text{GRPO}(\theta) = \mathbb{E}_{ x \sim \mathcal{D},\, \{y_i\}_{i=1}^G \sim \pi_{\theta_\text{old}}( \cdot | x) } \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min \left( \rho_{i,t}(\theta) \widehat{A}_{i,t},  \, \mathrm{clip} \left( \rho_{i,t}(\theta), 1 - {\varepsilon}, 1 + {\varepsilon}\right) \widehat{A}_{i,t} \right) \right],
\]
with token-level importance ratio
\[
\rho_{i,t}(\theta)=\frac{ \pi_{\theta} (y_{i,t} \mid x, y_{i,<t}) }{ \pi_{\theta_\text{old}} (y_{i,t} \mid x,y_{i,<t})},
\]
and group-relative normalized reward
\[
\widehat{A}_{i,t} = \widehat{A}_{i} = \frac{r(x, y_i) - \mathrm{mean} \left( \{ r(x, y_i) \}_{i=1}^G \right) }{ \mathrm{std} \left( \{ r(x, y_i) \}_{i=1}^G \right) }.
\]
This formulation already contains a kind of compression: every token in a response inherits the same group-normalized scalar advantage [2602.19327].

Compression-oriented extensions emerge because this baseline induces several mismatches. Rewards in reasoning RL are often sequence-level, yet correction is token-level. Long completions multiply token-level probability ratios across time, increasing variance. PPO-style hard clipping stabilizes updates but can discard useful signal. In binary-verification settings, many groups may also become ineffective when all sampled responses share the same label, causing zero or near-zero relative advantages [2602.19327], [2602.10048].

Another structural issue is redundancy. BPPO reports that, within the same prompt group, same-class completions often induce highly similar update directions, whereas correct-incorrect pairs provide more distinct contrastive signals [2605.28028]. This implies that standard GRPO may spend substantial computation updating on near-duplicate information. A related line of analysis argues that standard GRPO’s practical update rule is closer to a stale-policy gradient estimator than to a true current-policy gradient estimator, which further motivates simpler or more aggregated correction schemes [2508.02833].

The compression problem in C-GRPO therefore concerns both efficiency and statistical alignment: how to preserve the relative-comparison structure that makes GRPO attractive while eliminating unnecessary tokens, completions, variance sources, or model-side computation.

## 3. Compression by reward shaping: FGO and concise chain-of-thought

The clearest direct instantiation of compression-oriented GRPO is Fine-grained Group Policy Optimization (FGO), proposed for long CoT compression [2602.10048]. FGO starts from the observation that RL-trained reasoning models often generate unnecessarily verbose CoT trajectories containing redundant intermediate derivations, repeated self-verification, and overthinking. It defines compression operationally as reducing token length while preserving answer accuracy and reasoning capability [2602.10048].

FGO preserves the group-relative structure of GRPO but modifies reward construction inside the sampled group. In the paper’s formulation, vanilla reward is binary answer verification,
\[
r_i =
\begin{cases}
1, & \text{if } \hat{a}_i = a; \\
0, & \text{if } \hat{a}_i \neq a,
\end{cases}
\]
which in ordinary GRPO yields zero advantage whenever all responses in the group are either correct or incorrect. FGO addresses this by partitioning the group into a correct subgroup \(\mathcal G^+\) and an incorrect subgroup \(\mathcal G^-\), then shaping rewards within each subgroup using response length and entropy [2602.10048].

For the correct subgroup, the weight is
\[
\mathcal{W}^+ = \text{Softmax} \left[ \Big( \frac{ \text{mean} (L^+)}{L^+} \Big)^\alpha \times \Big( \frac{  \text{mean}(\mathcal{H}^+)}{\mathcal{H}^+} \Big) ^ \beta \right],
\]
and the shaped reward is
\[
R^+ = \mathcal{W}^+ \times \bar{R}^+,
\]
with \(\bar R^+=1\). Shorter-than-average and lower-entropy-than-average correct responses therefore receive larger positive weights [2602.10048]. FGO then replaces the standard GRPO advantage with a Dr.GRPO-style mean-centered version,
\[
A_{i,t} = r_i - \text{mean}(\{r_i\}_{i=1 }^{G}), \qquad r_i \in R,
\]
omitting the standard deviation denominator “for stability” [2602.10048].

The compression mechanism here is not token pruning. It is subgroup-relative reward shaping over whole responses. This distinction matters. FGO does not segment a trajectory into chunks and remove them; it changes the training signal so that shorter, still-correct reasoning paths become more rewarding. The paper reports strong evidence that this works: for Qwen2.5-Math-1.5B on MATH500, FGO improves accuracy from 65.6 under GRPO to 68.6 while reducing average length from 578 to 441 tokens and increasing ACT from 11.3 to 15.6; for DeepSeek-R1-Distill-Qwen-1.5B on MATH500, FGO improves accuracy from 51.0 to 56.4 while reducing length from 828 to 229 [2602.10048].

FGO also addresses a major GRPO degeneracy. Table 3 in the paper reports that invalid sample counts among 3,200 training samples drop from 1584, 1866, 2617, and 2444 under GRPO to 0 under FGO for four different 1.5B models [2602.10048]. This makes FGO particularly relevant to C-GRPO as a compression-aware reformulation of reward structure rather than an inference-time compression heuristic.

## 4. Compression by update selection: BPPO and pairwise prefix optimization

Binary Prefix Policy Optimization (BPPO) is a more aggressive form of C-GRPO in which compression is applied directly to the optimization payload [2605.28028]. The method preserves GRPO’s full-group sampling, reward computation, and advantage normalization, but updates only a compact subset of completions and only their prefixes.

The key empirical observation is that within-group same-class completions often induce highly similar update directions, while correct-incorrect pairs provide more informative contrastive signal [2605.28028]. Based on this, BPPO defines a binary update unit consisting of the shortest correct completion and the shortest incorrect completion from the same prompt group:
\[
i^*_+ = \arg\min_{i \in \mathcal{I}^+} \ell(o_i), \quad i^*_- = \arg\min_{i \in \mathcal{I}^-} \ell(o_i),
\]
with selected set
\[
\mathcal{S}(q)=\{i^*_+, i^*_-\}.
\]
The advantage \(\hat A_i\) is still computed using all \(G\) sampled completions. Only after full-group normalization does BPPO compress the update to this selected pair [2605.28028].

BPPO also applies prefix-focused optimization. Instead of updating over the full response length \(|o_i|\), it updates only the first \(n\) response tokens, where \(n\) is set to 50% of the average response length in the reported experiments [2605.28028]. The objective becomes
\[
J(\theta) = \mathbb{E}_{q,\{o_i\}_{i=1}^{G}\sim\pi_{\theta_{\mathrm{old}}(\cdot\mid q)} \Bigg[ \frac{1}{|\mathcal{S}(q)|} \sum_{i\in\mathcal{S}(q)} \Bigg( \frac{1}{n}\sum_{t=1}^{n} \Bigl[
\min\Big( \rho_{i,t}(\theta)\hat{A}_i,\, \mathrm{clip}(\rho_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\hat{A}_i \Big)
-\beta\,\mathbb{D}_{\mathrm{KL}}\!\left[\pi_\theta \,\|\, \pi_{\mathrm{ref}}\right]
\Bigr] \Bigg) \Bigg].
\]
This is a direct compression of both the completion axis and the token axis [2605.28028].

The efficiency gains are substantial. The paper reports up to \(6.08\times\) speedup over GRPO in the main tables, specifically on MATH with Llama3.2-3B-Instruct at \(G=16\), where training time falls from \(12638 \pm 878\) seconds to \(2079 \pm 73\) seconds [2605.28028]. Mean response length is reduced by approximately 30–50% without any explicit length penalty: on MATH at \(G=16\), average tokens drop from 374.83 to 185.20, a 50.6% reduction; on GSM8K at \(G=16\), from 262.64 to 147.25; on Geo3K at \(G=16\), from 3066.45 to 1665.13 [2605.28028].

BPPO therefore exemplifies a strong form of C-GRPO: full-group statistics, pair-only optimization, and prefix-only backpropagation. A plausible implication is that update-set compression can be effective when group redundancy is high, but BPPO also makes clear that this relies on binary correctness structure and on groups containing both classes.

## 5. Compression by sequence aggregation: SSPO, TEPO, and trajectory-level correction

A different C-GRPO lineage compresses token-level correction signals into a sequence-level statistic. The main motivation is the mismatch between sequence-level rewards and token-level importance weighting.

Soft Sequence Policy Optimization (SSPO) explicitly frames itself as a bridge between GMPO and SAPO within the GRPO family [2602.19327]. Its central idea is to aggregate softly gated token ratios geometrically across the sequence:
\[
\mathcal{J}_{\mathrm{SSPO}}(\theta) = \mathbb{E}_{x,\{y_i\}_{i=1}^G\sim \pi_{\theta_{\mathrm{old}}(\cdot \mid x)}} \left[ \frac{1}{G} \sum_{i=1}^{G} \left( \prod_{t=1}^{|y_i|} f_{i,t}\!\left(\rho_{i,t}(\theta)\right) \right)^{\frac{1}{|y_i|} \widehat{A}_{i} } \right],
\]
where \(f_{i,t}\) is the SAPO sigmoid gate and \(\rho_{i,t}\) is the token-level importance ratio [2602.19327]. The gate is
\[
f_{i,t}(x) = \sigma\left(\tau_{i,t}(x-1)\right)\frac{4}{\tau_{i,t}},
\qquad
\tau_{i,t} =
\begin{cases}
\tau_{\text{pos}}, & \widehat{A}_{i,t} > 0,\\
\tau_{\text{neg}}, & \widehat{A}_{i,t} \le 0.
\end{cases}
\]
SSPO therefore compresses an entire token-ratio trajectory into one softly attenuated sequence-level weight [2602.19327]. This is directly relevant to C-GRPO if compression is understood as robust sequence summarization of off-policy corrections.

TEPO pursues a related objective using a length-normalized sequence importance ratio derived from the Markov factorization of autoregressive sequence likelihood [2510.09369]:
\[
\mathcal{IS}_i(\theta) =
\left(
\frac{\pi_\theta(y_i\mid x)}
{\pi_{\theta_{\text{old}}}(y_i\mid x)}
\right)^{1/|y_i|}
=
\exp\left(
\frac{1}{|y_i|}
\sum_{t=1}^{|y_i|}
\log
\frac{\pi_\theta(y_{i,t}\mid x, y_{i,<t})}
{\pi_{\theta_{\text{old}}}(y_{i,t}\mid x, y_{i,<t})}
\right).
\]
Its core loss is
\[
L(\theta) = \frac{1}{\sum_{i=1}^{G} |o_i|} \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} \min\Big( \mathcal{IS}_i(\theta)\hat A_{i,t}, \operatorname{clip}(\mathcal{IS}_i(\theta),1-\varepsilon,1+\varepsilon)\hat A_{i,t} \Big),
\]
which replaces tokenwise importance correction with a single sequence-level geometric-mean ratio [2510.09369]. TEPO’s backward pass then redistributes the sequence-level signal across tokens through the Markov structure.

This suggests a “compressed statistic” view of C-GRPO: rather than store or optimize with all token-level ratios directly, one can compress them into a single trajectory-level or sequence-level factor before backpropagation. That perspective is reinforced by "On the Theory and Practice of GRPO: A Trajectory-Corrected Approach with Fast Convergence" [2508.02833], which argues that standard practical GRPO’s token-level importance correction estimates the gradient at the old policy rather than the current one. The paper proposes trajectory-level importance-corrected GRPO (TIC-GRPO), replacing token-level ratios with
\[
w'(s_T^{(i)},\theta,\theta_{\mathrm{old}}) :=
\frac{\Pr_\theta(s_T^{(i)}\mid s_0^{(i)})}
{\Pr_{\theta_{\mathrm{old}}}(s_T^{(i)}\mid s_0^{(i)})},
\]
and claims this yields an unbiased estimate of the current-policy gradient while preserving the critic-free structure [2508.02833]. A plausible implication is that trajectory-level correction is a principled compression of token-level importance information, not merely an engineering shortcut.

## 6. Compression by rollout reduction, co-distillation, and constraints

Several additional papers broaden the meaning of C-GRPO beyond output length and token aggregation.

"It Takes Two: Your GRPO Is Secretly DPO" argues that GRPO can be reframed as a contrastive objective and that a two-rollout version, 2-GRPO, can match 16-GRPO while using only \(1/8\) of the rollouts and reducing training time by over 70% [2510.00977]. In the pairwise case, if one rollout is positive and one negative, the advantages collapse to
\[
A^+ = 1,\quad A^- = -1,
\]
and otherwise both are zero. The paper reports 0.15 million generated rollouts for 2-GRPO versus 1.2 million for 16-GRPO, with wall-clock reductions of roughly 74–84% depending on model and dataset [2510.00977]. This is a strong form of rollout compression.

CoDistill-GRPO addresses a different bottleneck: small models often fail under standard GRPO because sparse reward provides too little signal on hard tasks [2605.08873]. Its small-model objective adds an on-policy KD reward
\[
\widetilde{r}_i
=
r(q,o_i)
+
\alpha \cdot \frac{1}{N}\sum_{t=1}^{N}
\log\left(
\frac{\pi_{\theta}(o_{i,t}\mid q, o_{i,<t})}
{\pi_{\phi}(o_{i,t}\mid q, o_{i,<t})}
\right),
\]
while the large model is updated off-policy using rollouts generated by the small model [2605.08873]. The result is a form of systems-level compression: the expensive model no longer bears the dominant rollout-generation burden. The paper reports an approximate 18% speedup for the larger model and substantial gains for the smaller model, including over 11.6 percentage points over the base model and an additional 6.0 points over GRPO on Minerva for Qwen2.5-Math-1.5B [2605.08873].

Constrained GRPO is relevant when “compression” is expressed as an explicit constraint rather than a soft preference [2602.05863]. The paper shows that naive scalarization of reward and constraint before group normalization introduces implicit reweighting:
\[
A_{\mathrm{ScRew}}
=
\sum_{j=0}^{K} \frac{\lambda_j \sigma_j}{\sigma_{R_s}} Z_j,
\]
so the effective coefficient on each component is not \(\lambda_j\) but \(\lambda_j \sigma_j / \sigma_{R_s}\) [2602.05863]. It proposes scalarizing standardized component advantages instead:
\[
A_{\mathrm{ScAdv}} = \lambda_R Z_R - \sum_{k=1}^{K} \lambda_k Z_{C_k}.
\]
This is highly relevant to compression-aware GRPO because token budget, latency, bandwidth, or compute caps can be encoded as indicator costs. The paper’s main lesson is that if compression is treated as a constraint, a C-GRPO implementation should scalarize advantages rather than rewards, or the intended trade-off can be corrupted by variance mismatch [2602.05863].

## 7. Empirical patterns, controversies, and open questions

Across these papers, several empirical patterns recur. Compression-oriented GRPO variants often improve token efficiency, sometimes while preserving or improving accuracy. FGO and BPPO both report shorter outputs with competitive or improved benchmark performance [2602.10048], [2605.28028]. Pairwise and subset-based methods report large wall-clock savings, especially when rollout generation dominates cost [2510.00977], [2605.28028]. Sequence-level aggregation methods are typically motivated by better alignment between reward granularity and correction granularity, though some of their claims remain more conceptual than fully numerically established in the visible text [2602.19327].

There are also important controversies and caveats. First, several methods rely on binary correctness signals, which may not generalize directly to partial-credit or open-ended tasks [2602.10048], [2605.28028], [2510.00977]. Second, the distinction between compression of behavior and compression of optimization is often blurred. BPPO, for example, reduces response length without an explicit length penalty by not reinforcing suffixes, whereas FGO explicitly reweights rewards toward shorter trajectories [2605.28028], [2602.10048]. Third, some formulations contain minor inconsistencies between prose and equations, especially in incorrect-group weighting or clipping notation, which complicates direct transplantation [2602.10048], [2508.02833].

A broader theoretical tension concerns the fidelity of token-level PPO-style correction in GRPO. One line argues that standard GRPO’s token-level correction does not estimate the true current-policy gradient and that simplified stale-policy updates can perform comparably [2508.02833]. Another line emphasizes off-policy GRPO and clipped surrogates with theoretical reward-improvement guarantees, showing that both on-policy and off-policy GRPO can improve reward under appropriate total-variation control [2505.22257]. This suggests that the “right” compression of importance information remains unsettled: pairwise reduction, trajectory-level correction, geometric aggregation, and stale-policy simplification each have distinct justifications.

A final open issue is that “C-GRPO” itself remains non-standardized. Current evidence supports the existence of a coherent design space rather than a canonical named algorithm. This suggests that future work may stabilize around a narrower definition, but at present the term is best understood as a descriptive umbrella for compression-aware GRPO-family methods rather than a single fixed objective.

## 8. Synthesis

Compression Group Relative Policy Optimization refers most plausibly to GRPO-family methods that retain group-relative credit assignment while compressing some part of the training signal or update process. The literature supports four major instantiations of this idea.

First, reward shaping can compress behavior, as in FGO, by teaching policies to prefer shorter, still-correct reasoning trajectories [2602.10048]. Second, update selection can compress the optimization payload, as in BPPO and 2-GRPO, by retaining only informative pairs or prefixes [2605.28028], [2510.00977]. Third, sequence aggregation can compress token-level correction into robust trajectory-level statistics, as in SSPO, TEPO, and TIC-GRPO [2602.19327], [2510.09369], [2508.02833]. Fourth, co-distillation and constrained formulations can compress systems cost or impose explicit compression budgets through auxiliary objectives or Lagrangian costs [2605.08873], [2602.05863].

Taken together, these works suggest that the essential design principle of C-GRPO is not mere shrinking of outputs. It is the replacement of overly fine-grained, redundant, or poorly aligned policy-update signals with compact surrogates that preserve the relative-comparison logic of GRPO. In that sense, C-GRPO is best viewed not as a single algorithm but as a developing research program inside critic-free, group-relative policy optimization.

Source: https://www.emergentmind.com/topics/compression-group-relative-policy-optimization-c-grpo