---
title: 'PS-GRPO: Paraphrased Set Group Relative Policy Optimization'
url: https://www.emergentmind.com/topics/paraphrased-set-group-relative-policy-optimization-ps-grpo
type: topic
---

# PS-GRPO: Paraphrased Set Group Relative Policy Optimization

Paraphrased Set Group Relative Policy Optimization (PS-GRPO) is a reinforcement learning (RL) algorithmic framework developed to improve consistency, alignment, and performance of language models (LLMs) and multi-output systems in scenarios where semantically equivalent inputs—such as paraphrased queries—should yield consistent, reliable outputs. By extending the principles of Group Relative Policy Optimization (GRPO), PS-GRPO leverages group-wise preference aggregation, consistency rewards, and scalable RL objectives to address core challenges in retrieval-augmented generation, reasoning, and other structured prediction domains.

## 1. Conceptual Foundations and Motivation

PS-GRPO generalizes GRPO to settings where multiple semantically equivalent input variations, termed a *paraphrased set*, are provided simultaneously. Standard GRPO computes the group-relative advantage by normalizing rewards obtained from a group of candidate outputs sampled for a single context, adjusting the policy toward higher-reward members while regularizing against excessive deviation from a reference policy. PS-GRPO expands this mechanism by explicitly organizing outputs not just for a single context, but across paraphrase sets—thereby directly targeting consistency in the face of linguistic, structural, or system-induced variability [2510.04392].

The motivation for PS-GRPO arises from the requirement that AI systems, especially those deployed in safety-critical applications (e.g., healthcare, finance, legal search), must yield outputs that are invariant to minor input perturbations such as paraphrasing. Empirical observations show that conventional LLMs and retrieval-augmented generators can produce divergent answers for inputs that are semantically equivalent, undermining reliability and user trust.

## 2. Algorithmic Structure and Policy Objective

The PS-GRPO objective is rooted in the GRPO family, which fundamentally combines two terms in its learning objective: a normalized group-relative advantage signal and a penalty term that reduces divergence from a reference policy. In the paraphrased set extension, the procedure is as follows:

- For a canonical query $q_0$, a set $\mathcal{P}(q_0) = \{p_1, ..., p_n\}$ of paraphrased variants is generated.
- For each paraphrase $p_i$, the model generates $g$ output samples $\{o_{i1}, ..., o_{ig}\}$.
- The group similarity reward $r_{ij}$ for output $o_{ij}$ is computed by averaging its similarity to all outputs with the other paraphrases:

$$
r_{ij} = \frac{1}{(n-1)g} \sum_{u \neq i} \sum_{m=1}^{g} \mathrm{sim}(o_{ij}, o_{um})
$$

where $\mathrm{sim}(\cdot, \cdot)$ is typically a lexical similarity metric such as BLEU; this can be extended to semantic metrics in future work.

- The group-relative advantage for output $o_{ij}$ is then normalized within its paraphrase group:

$$
\hat{A}_{ij} = \frac{r_{ij} - \mu_i}{\sigma_i}, \quad \mu_i = \frac{1}{g} \sum_{j=1}^{g} r_{ij}, \;\; \sigma_i = \sqrt{\frac{1}{g} \sum_{j=1}^{g} (r_{ij} - \mu_i)^2}
$$

- The optimization objective takes the PPO-style clipped update form:

$$
L_{PS\text{-}GRPO}(\theta) = \frac{1}{g} \sum_{j=1}^{g} \sum_{t=1}^{|o_{ij}|} [\min(\rho_{ij,t} \hat{A}_{ij}, \mathrm{clip}(\rho_{ij,t}, 1-\epsilon, 1+\epsilon) \hat{A}_{ij})] - \beta \cdot D_{KL}(\pi_{\theta}(\cdot|q) \Vert \pi_{\mathrm{ref}}(\cdot|q))
$$

where $\rho_{ij,t}$ is the token-level probability ratio relative to the reference policy, and $\beta$ tunes regularization toward the reference.

## 3. Preference Aggregation and Consistency Reward Construction

Unlike logarithmic pooling (RLHF), which exponentially tilts the reference policy toward higher-reward candidates, GRPO—and by extension PS-GRPO—utilizes an inverse-linear weighting:

$$
\pi_{\theta}(o|q) = g\left(\frac{\mathcal{P}_G(o|\pi_{\theta}(\cdot|q), q) - E_{o'}[\mathcal{P}_G(o'|\pi_{\theta}(\cdot|q), q)]}{\beta}\right) \cdot \pi_{\mathrm{ref}}(o|q), \quad g(x) = \frac{1}{1-x}
$$

This means outputs with above-average group preference receive an amplified probability, but the amplification is modulated to avoid instability. In PS-GRPO, this group preference is specifically engineered to encourage output similarity across paraphrases [2502.18548, 2510.04392].

The reward function can flexibly utilize similarity metrics (e.g., BLEU, ROUGE), stepwise reasoning agreement, or LLM-based factual consistency judgments. The choice of normalization, reward scaling, and penalty term (reverse vs. direct KL) can be tuned to interpolate between different aggregation regimes.

## 4. Computational Scalability and Approximation

Direct computation of group similarity rewards for every ($n$, $g$) pair entails quadratic complexity. To ensure scalability, PS-GRPO implements subsampled reward estimation:

$$
\tilde{r}_{ij} = \frac{1}{\kappa s} \sum_{u \in K} \sum_{m \in S_k} \mathrm{sim}(o_{ij}, o_{um})
$$

where $K \subset \{1,\dots,n\}$ and $S_k \subset \{1,\dots,g\}$ are randomly chosen subsets ($\kappa \ll n$, $s \ll g$). This approach maintains a strong training signal while constraining computation during large-scale QA or generation tasks [2510.04392].

## 5. Empirical Performance and Practical Applications

PS-GRPO yields marked improvements in output consistency on short-form, multi-hop, and long-form QA datasets. For example, on TriviaQA, consistency metrics rise from 53% (standard RAG) to 87% under PS-GRPO-driven Con-RAG. The improvement holds for both lexical and information-level consistency, as measured by similarity scores and LLM-based judges.

Importantly, PS-GRPO also exhibits accuracy gains, often in the absence of explicit ground-truth rewards. This suggests that the consistency reward has a regularizing effect analogous to data augmentation, making the generator more robust to input and retrieval variability.

## 6. Alignment Objective, Parametric Dependence, and Extensions

PS-GRPO maintains the core GRPO alignment objective: maximizing expected normalized group advantage—here, consistency—and penalizing policy drift from reference. The form of the aggregate preference and stationary policy depends on hyperparameters including the regularization constant $\beta$, the scale of similarity rewards, and group size. Modifications such as direct versus reverse KL penalties, omission of scale normalization, or dynamic entropy weighting (cf. [2508.04349]) further modulate sensitivity to reward differences and regularization strength.

Key extensions include:
- Hybrid consistency-semantic reward signals to encourage deeper semantic agreement beyond surface overlap.
- Joint retriever-generator optimization to mitigate inconsistencies arising from evidence variability.
- Application to other structured tasks: hyperparameter optimization [2509.17105], continuous control [2507.19555], and reasoning in security [2507.03051] and wireless systems [2509.15006].

## 7. Limitations, Challenges, and Future Directions

While PS-GRPO robustly enhances consistency and accuracy, several limitations are noted:
- Trade-off exists between reward fidelity and computational tractability; heavy subsampling may underrepresent critical paraphrase divergences.
- Lexical reward metrics can over-penalize stylistically valid paraphrases; future research is needed on semantic and factual alignment scores.
- Policy stability depends critically on tuning $\beta$ and group size; excessive regularization can impede learning, while weak regularization can cause overfitting to surface form.

A plausible implication is that further refinements involving dynamic reward weighting, adaptive group formation, and integration with off-policy robustness principles [2509.24203] will be required for optimal deployment in open-ended, retrieval-intensive and safety-critical scenarios.

---

PS-GRPO constitutes a principled, scalable paradigm for improving output consistency and reliability in multi-input, multi-output systems. By leveraging group similarity rewards and GRPO-style normalized advantages, it provides a robust solution to the persistent challenge of aligning model behavior across paraphrased or variable inputs, with broad applicability across language, reasoning, and control domains [2510.04392; 2502.18548].

Source: https://www.emergentmind.com/topics/paraphrased-set-group-relative-policy-optimization-ps-grpo