---
title: Consistency Group Relative Policy Optimization (Con-GRPO)
url: https://www.emergentmind.com/topics/consistency-group-relative-policy-optimization-con-grpo
type: topic
---

# Consistency Group Relative Policy Optimization (Con-GRPO)

Consistency Group Relative Policy Optimization (Con-GRPO) is a reinforcement learning (RL) algorithmic framework developed to directly optimize consistency objectives across groups of semantically equivalent inputs. The primary context of its application is language generation systems, particularly Retrieval-Augmented Generation (RAG), where ensuring that paraphrased queries yield consistent (i.e., stable in informational content) answers is essential for reliability, trust, and compliance in high-stakes settings. Con-GRPO instantiates Group Relative Policy Optimization (GRPO), where advantage estimation and optimization occur not for individual samples, but in a groupwise context with rewards reflecting cross-sample agreement, thereby enabling direct control of consistency properties [2510.04392].

## 1. Formal Setting, Definitions, and Notation

Con-GRPO is typically instantiated in RAG architectures, comprising:
- A retriever $R$ that maps a query $q$ to a document set $R(q)=\mathcal{D}(q)\subset\mathcal{D}$.
- A generator (parametric policy) $\pi_\theta$, which yields a distribution over output sequences $y$ conditional on the query and retrieved documents:
  $$
  \pi_\theta(y|q) = \pi_\theta(y|q,\;R(q))
  $$
- A paraphrase set (group) $G = \{q_1, \dots, q_n\} = \mathcal{P}(q_0)$, representing $n$ semantically equivalent variants of canonical query $q_0$.

For each paraphrase $q_i$, $g$ rollouts are sampled, generating $\{o_{i,1}, \ldots, o_{i,g}\}$, with $o_{i,j}\sim\pi_\theta(\cdot|q_i)$ for $i=1,\ldots,n$, $j=1,\ldots,g$. 

The principal consistency goal is for all outputs corresponding to any paraphrase in $G$ to convey the same core information, regardless of input phrasing or retriever variability.

## 2. Group Similarity Rewards and Computation

Central to Con-GRPO is the *group similarity reward*, which incentivizes statistical agreement among the outputs for all members of the paraphrase set. Pairwise similarity between outputs is measured using a token-level function, most commonly BLEU-$k$ (unigram for short-form, bigram for long-form QA):

\[
r_{i,j} = \frac{1}{(n-1)\,g} \sum_{u\neq i}^{n} \sum_{m=1}^{g} \operatorname{sim}\bigl(o_{i,j},\,o_{u,m}\bigr)
\]

Each rollout's reward is its average similarity to all rollouts from other paraphrases in the group. For short-form QA where a reference $y^\star$ exists, an accuracy term $\mathrm{Acc}(o_{i,j}, y^\star)$ (e.g., token F1 or exact match) is added, forming the final reward:

\[
r_{i,j}^{\mathrm{final}} = \alpha\,r_{i,j}^{\mathrm{cons}} + \gamma\,\mathrm{Acc}(o_{i,j},\,y^\star)
\]

where hyperparameters $\alpha, \gamma \ge 0$ (typically $\alpha=\gamma=1$ for short-form, $\gamma=0$ for long-form/open-ended).

To address the $O(n^2g^2)$ cost of all-pairs group reward computation, a scalable unbiased approximation samples only a subset of other paraphrases ($\kappa \ll n-1$) and rollouts ($s \ll g$):

\[
\tilde r_{i,j} = \frac{1}{\kappa\,s} \sum_{u\in K} \sum_{m\in S_u} \operatorname{sim}(o_{i,j}, o_{u,m})
\]
with $K\subset\{1,\ldots,n\}\setminus\{i\}$ and $S_u\subset\{1,\ldots, g\}$.

In practice, $n=6$, $g=4$, $\kappa=3$, $s=1$ suffice, resulting in linear computational scaling.

## 3. Policy Optimization and GRPO Objective

Rather than estimate a state-value critic, Con-GRPO employs a within-group normalization for advantage estimation. For each paraphrase $q_i$:

- Compute the reward mean $\mu_i$ and standard deviation $\sigma_i$:
  $$
  \mu_i = \frac{1}{g}\sum_{j=1}^g r_{i,j}, \quad
  \sigma_i = \sqrt{\frac{1}{g}\sum_{j=1}^g (r_{i,j} - \mu_i)^2}
  $$
- The normalized advantage is
  $$
  \hat{A}_{i,j} = \frac{r_{i,j} - \mu_i}{\sigma_i + \epsilon}
  $$
  for numerical stability.

The GRPO surrogate loss (PPO-style) is:

\[
\mathcal{L}_\mathrm{GRPO}(\theta) = \frac{1}{ng} \sum_{i=1}^n \sum_{j=1}^g \sum_{t=1}^{|o_{i,j}|} 
\min\left(\rho_{i,j,t} \hat{A}_{i,j}, \operatorname{clip}(\rho_{i,j,t}, 1-\epsilon, 1+\epsilon)\hat{A}_{i,j}\right)
- \beta\,\mathrm{KL}\left(\pi_\theta(\cdot|q_i)\Vert\pi_{\mathrm{ref}}(\cdot|q_i)\right)
\]

where $\rho_{i,j,t}$ is the (possibly token-level) importance weight, $\epsilon$ the clipping parameter, and $\beta$ a KL-regularization coefficient to avoid policy drift.

In effect, this structure:
- Increases the probability of outputs whose normalized agreement within their paraphrase group is high.
- Penalizes significant divergence from the reference (pre-trained) policy.

## 4. Training Procedure, Implementation, and Hyperparameters

The Con-GRPO procedure is organized as follows:
1. For each batch, sample a set of canonical queries.
2. For each canonical query, generate $n$ paraphrases.
3. Retrieve documents and sample $g$ rollouts per paraphrase.
4. Compute approximate group similarity rewards per rollout.
5. Normalize rewards within paraphrase groups to compute advantages.
6. Accumulate the clipped GRPO loss and update parameters via backpropagation.

Key hyperparameters:
- Paraphrase set size: $n=6$
- Rollouts per paraphrase: $g=4$
- BLEU-$1$ for short-form, BLEU-$2$ for long-form
- Reward weights: $\alpha=1$, $\gamma=1$ (short-form), $\gamma=0$ (open-ended)
- KL penalty: $\beta=0.0$ (supervised), $\beta=0.05$ (open-ended)
- Optimizer: AdamW, learning rate $1\mathrm{e}{-6}$
- Decoding temperature at training matches inference (commonly $T=0.0$ for deterministic rollouts)
- Batch size: 1–2 canonical queries per GPU (dependent on model/memory)

## 5. Empirical Outcomes and Benchmarks

Con-GRPO, instantiated as "Information Consistent RAG" (Con-RAG), demonstrates pronounced consistency and accuracy gains relative to standard RAG and strong RL baselines (e.g., DRAG, CoT-RAG, SFT):

**Short-form QA (LLaMA-3.1-8B, TriviaQA):**
| Metric                      | RAG Baseline | Con-RAG |
|-----------------------------|--------------|---------|
| End-to-end lexical consistency (%)  | 53.0         | 87.3    |
| End-to-end LLM-judge consistency (%)| 77.8         | 91.3    |
| Generator lexical consistency (%)   | 67.3         | 91.2    |
| Generator LLM-judge consistency (%) | 88.5         | 93.0    |
| Exact Match accuracy (%)            | 56.0         | 77.0    |
| Token F1 (%)                        | 66.1         | 81.0    |

**Long-form QA (ELI5, LLaMA-3.1-8B):**
| Metric                       | RAG Baseline | Con-RAG |
|------------------------------|--------------|---------|
| End-to-end lexical consistency (%)   | 8.6          | 14.6    |
| End-to-end LLM-judge consistency (%) | 62.8         | 72.7    |
| Generator lexical consistency (%)    | 15.1         | 21.7    |
| Generator LLM-judge consistency (%)  | 74.2         | 80.8    |
| ROUGE accuracy                      | 21.9         | 24.2    |
| LLM-judge accuracy (%)              | 74.0         | 78.0    |

Con-GRPO continues to outperform supervised-fine-tuning (SFT) even when ground truth is unavailable, confirming the efficacy of group similarity rewards for open-ended or reference-free tasks [2510.04392].

## 6. Relationship to Broader GRPO Paradigm and Extensions

Con-GRPO is a specialization of GRPO for information consistency, leveraging group reward computation and normalization within paraphrase sets. Related GRPO variants extend the paradigm to other objectives:
- **Constrained GRPO** imposes explicit behavioral constraints via Lagrangian relaxation and scalarized advantage construction, outperforming naive scalarization approaches in both theory and practice [2602.05863].
- **Consensus GRPO** distills Minimum Bayes Risk decoding into a policy optimized using only groupwise consensus (e.g., BLEURT) as utility, eliminating reliance on gold references or preference labels [2602.03102].
- **Continuous Control Con-GRPO** applies group-based normalization to trajectory clusters and state-aware credit assignment for high-dimensional, continuous-action tasks [2507.19555].

These extensions maintain the core principle of intra-group normalized advantage estimation, with variations in reward construction and applicability.

## 7. Limitations and Future Directions

While Con-GRPO has demonstrated clear empirical gains in consistency and accuracy across QA and RAG tasks, limitations and open research directions include:
- Reward functions are dependent on lexical similarity (e.g., BLEU), which may not fully capture semantic agreement.
- Approximate reward computation, while efficient, introduces stochastic variance not present in exhaustive all-pairs evaluation.
- Extension to more diverse and challenging paraphrase sets, dialogue, non-English queries, and non-retrieval settings remains to be explored.
- Integration of richer semantic similarity metrics or human-aligned utility functions to minimize the risk of overfitting to lexical form rather than true informational content.

The framework provides a reproducible and theoretically grounded approach for aligning large-scale neural generation systems with consistency-centric deployment constraints [2510.04392].

Source: https://www.emergentmind.com/topics/consistency-group-relative-policy-optimization-con-grpo