Commonsense Persona-Grounded Dialogue Challenge 2025

• CPDC 2025 is a challenge that benchmarks dialogue systems on integrating commonsense reasoning with persona-grounded interactions.
• It introduces innovative evaluation methods and datasets to measure coherence, consistency, and naturalness in conversational AI.
• Preliminary insights suggest potential improvements in handling ambiguities, enhancing response relevance, and ensuring realistic persona simulation.

Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that replaces value-function learning with group-normalized advantage estimation. Originally formulated to address stable post-training of large language and vision models under supervised or verifiable reward regimes, GRPO enforces policy improvement by standardizing rewards among a small group of rollouts per prompt, and combining this with a reverse-Kullback–Leibler (KL) penalty with respect to a reference policy. Its stationary solutions implement a rational, group-relative aggregate of preferences fundamentally distinct from the exponential pooling characteristic of standard RLHF methods such as reward model fine-tuning.

1. Reward Preference Model and Advantage Normalization

GRPO’s core innovation is its design of an advantage function computed by shift-and-scale normalizing the rewards within a sampled group. For each context $q$, a group of $G$ outputs $\{o_1, ..., o_G\}$ is drawn independently from a reference or old policy, and each output is assigned a reward $r_i = r(o_i|q)$. The group-relative advantage for each candidate is:

$$A_i = \frac{ r_i - \overline{r} }{ \mathrm{std}(r_1, ..., r_G) },$$

where $\overline{r}$ is the group mean. This normalization induces scale- and shift-invariance and allows for robust learning signals without an explicit value network.

The group-relative preference function for a candidate $o$ given a group $\{o'_j\}$ is

$$\mathcal{P}_G(o|\{o'_j\}, q) = \mathbb{E}[A_1|o_1 = o, o'_j],$$

and aggregating over sampled groups yields the policy-level reward-preference term.

Remarkable cases:

• For $G=2$, this reduces to the pairwise comparison probability.
• As $G \rightarrow \infty$, the advantage approaches normalized difference in expected reward scaled by group reward variance (Vojnovic et al., 25 Feb 2025).

2. Policy Objective, Reverse KL Penalty, and Stationary Solutions

The full (unclipped) GRPO optimization objective, averaged over groups from $\pi_{\rm old}$, is:

$$\mathcal{J}_{GRPO}(\theta) = \mathbb{E}_{q} \mathbb{E}_{\{o_i\}} \left[ \frac{1}{G}\sum_{i=1}^G \left( \frac{ \pi_\theta(o_i|q) }{ \pi_{\rm old}(o_i|q) } A_i - \beta D_i(\theta) \right) \right],$$

where $D_i(\theta)$ approximates the reverse KL by importance sampling: $$D_i(\theta) = \frac{ \pi_{\rm ref}(o_i|q) }{ \pi_\theta(o_i|q) } - \log \frac{ \pi_{\rm ref}(o_i|q) }{ \pi_\theta(o_i|q) } - 1.$$

This loss encourages boosting policy probability on relatively preferred outputs while penalizing deviation from the reference; in the stationary limit this penalty converges to the reverse KL $\mathrm{KL}(\pi_{\rm ref}\|\pi_\theta)$ (Vojnovic et al., 25 Feb 2025).

The stationary policy for each context solves the fixed-point equation:

$$\Bigg( 1 - \frac{ \mathcal{P}_G(o|\pi, q) - \mathbb{E}_{o'}[ \mathcal{P}_G(o'|\pi, q) ] }{ \beta } \Bigg) \pi(o|q) = \pi_{\rm ref}(o|q)$$

or, equivalently,

$$\pi(o|q) = \frac{1}{ 1 - \frac{ \mathcal{P}_G(o|\pi, q) - \bar{\mathcal{P}}_G(\pi, q) }{ \beta } } \pi_{\rm ref}(o|q)$$

Contrast: In RLHF, stationary policies are proportional to $\pi_{\rm ref} \exp( r_\phi / \beta )$, forming a logarithmic pooling; in GRPO, the aggregation is rational-function based and group-relative (Vojnovic et al., 25 Feb 2025). For direct KL penalties, GRPO can be made to reduce to the RLHF form.

3. Special Cases and Decision Rules

a. Group Size Two (Pairwise)

For $G = 2$, the advantage becomes a sign function, and the stationary policy satisfies a quadratic equation, yielding explicit expressions. For binary output problems, the stationary policy assigns probabilities incorporating the pairwise margin between alternatives and the regularization parameter (Vojnovic et al., 25 Feb 2025). In this limit, GRPO closely mimics preference-learning, operating essentially as a pairwise comparator.

b. Large-Group Limit

For $G \rightarrow \infty$, the group-relative reward-preference simplifies to a normalized value function difference, and the fixed-point policy depends on the empirical variance of the group. Explicit formulas for binary tasks show that as $\beta$ increases, the final policy interpolates between the reference and a uniform maximization of relative reward.

c. Direct KL and Shift-Only Variants

If the reverse KL is replaced with a direct KL, or if group rewards are only shifted and not scaled, the aggregation matches RLHF-style logarithmic pooling. The use of reward scaling is unique to standard GRPO and central to its distinct alignment properties (Vojnovic et al., 25 Feb 2025).

4. Empirical and Algorithmic Properties

GRPO is explicitly batch- and group-based, admitting efficient implementation via Monte Carlo sampling and expectation over old-policy outputs. Its normalization scheme confers robustness to reward scaling and reduces the need for reward model calibration. The reverse KL penalty prevents unchecked divergence from the reference and is theoretically grounded as a necessary constraint for stationary existence.

Training dynamics:

• GRPO can amplify the success probability of the reference policy under mild assumptions on group size and the KL regularization constant (β).
• In practical instantiations (e.g., DeepSeek-R1-Zero), GRPO has shown stable training even with verifiable or hard-reward signals, and the preferred group size may be as small as two with only minor degradation in exploration or estimation efficiency (Wu et al., 1 Oct 2025).

5. Practical Implementation and Comparisons

The algorithmic pipeline for GRPO requires, for each prompt:

1. Sampling a group of rollouts.
2. Computing group-normalized advantages.
3. Updating the policy with a likelihood-ratio-weighted advantage and a reverse KL anchor to the reference.

Pseudocode for a single update step follows:

for q in Q: outputs = [sample_from_policy_old(q) for _ in range(G)] rewards = [reward_fn(o, q) for o in outputs] mean_r, std_r = np.mean(rewards), np.std(rewards) advantages = [(r - mean_r)/std_r for r in rewards] # Compute policy ratios and reverse KL # Update policy via clipped surrogate and anchor to reference

Specialized extensions—such as off-policy variants, diversity-aware reweighting, or multi-answer extensions—retain the core normalization and relative-comparison mechanism.

6. Theoretical and Conceptual Significance

GRPO departs fundamentally from exponential preference pooling by using group-relative normalization and enforcing a rational form of reward aggregation. This leads to stationary solutions with distinct exploration–exploitation and regularization tradeoffs. The explicit incorporation of group-level statistics confers robustness when rewards are poorly calibrated or stochastic. Comparative and limiting-case analyses demonstrate that GRPO encompasses, in special cases, well-known RLHF-style objectives but can also reproduce pairwise-preference learning as a special case for $G=2$ (Vojnovic et al., 25 Feb 2025).

Summary Table: GRPO vs. RLHF Aggregation

Mechanism	Aggregation Rule	Limit Behavior
GRPO (std)	Rational function (1/(1-x))	Pairwise/raw at G=2; mean-variance at large G
RLHF (log)	Exponential function (exp(x))	Reduces to softmax over reward differences
GRPO (KL direct)	Exponential / log-pooling	RLHF objective

GRPO thus stands as a principled alternative to standard RLHF, balancing preference aggregation, regularization, and robust policy improvement through its group-relative reward normalization and rational pooling mechanism. References: (Vojnovic et al., 25 Feb 2025, Wu et al., 1 Oct 2025, Mroueh, 9 Mar 2025).