---
title: 'GiGPO: Group-in-Group Policy Optimization'
url: https://www.emergentmind.com/topics/group-in-group-policy-optimization-gigpo
type: topic
---

# GiGPO: Group-in-Group Policy Optimization

Group-in-Group Policy Optimization (GiGPO) is a reinforcement learning (RL) algorithm designed to address the challenges of credit assignment in long-horizon large language model (LLM) agent environments. GiGPO introduces a hierarchical mechanism for estimating advantages at both the episodic and step levels, leveraging group-based statistics to yield fine-grained credit signals without requiring critics or auxiliary models. The approach preserves the advantages of prior group-based RL methods—namely, critic-free design, low memory overhead, and stable convergence—while overcoming their limitations in environments with sparse and delayed rewards, as commonly encountered in complex LLM agent tasks [2505.10978].

## 1. Background and Motivation

Long-horizon agent environments, such as ALFWorld (up to 50 steps, 20K tokens per episode) and WebShop (complex web browsing, multi-turn, 1.1 M products), challenge traditional RL algorithms due to severe reward sparsity and delayed feedback—often only a single success or failure signal per episode. In these settings, standard group-based RL algorithms such as RLOO and GRPO compute trajectory-level returns within groups and estimate a single relative advantage per trajectory by normalizing each return against its peers. While effective and stable for single-turn tasks, these methods collapse all intermediate actions into a single scalar and fail to provide step-wise credit assignment, impeding optimization in settings where agent-environment interactions span many steps and optimal behaviors may hinge on individual action quality at specific junctures. GiGPO was developed to address this limitation by enabling fine-grained (per-step) credit signals without compromising group RL's critic-free, memory-efficient properties.

## 2. Two-Level Advantage Estimation Architecture

GiGPO operationalizes a hierarchical advantage estimation process encompassing two nested levels:

### 2.1 Episode-Level (Macro) Relative Advantage

At the highest level, GiGPO samples $N$ complete trajectories $\{\tau_i\}_{i=1}^N$ under the same task $x$ and identical initial state $s_1^{(1)}=...=s_1^{(N)}$. Each trajectory’s return is defined as $R(\tau_i)=\sum_{t=1}^{T}r_t^{(i)}$ (or $R(\tau_i) \in \{0,1\}$ in fully terminal-reward tasks). These are grouped:
$$
G^E = \{ (\tau_i, R(\tau_i)) \}_{i=1}^N
$$
The episode-level ("macro") relative advantage for trajectory $\tau_i$ is:
$$
A^E(\tau_i) = \frac{R(\tau_i) - \mathrm{mean}\{R(\tau_j)\}}{F_{\mathrm{norm}}\left(\{R(\tau_j)\}\right)}
$$
where $F_{\mathrm{norm}}$ is typically the group's standard deviation, or can be set to $1$ for the unbiased Leave-One-Out variant.

### 2.2 Step-Level (Micro) Relative Advantage via Anchor-State Grouping

Within the same group, anchor-state grouping is performed by hashing all states encountered across $\{\tau_i\}$ to create a set $\mathcal{U} = \{\tilde s_1, ..., \tilde s_U\}$ of unique states. For each anchor state $\tilde s$, all occurrences $(s_t^{(i)}, a_t^{(i)}, r_t^{(i)})$ with $s_t^{(i)}=\tilde s$ are collected:
$$
G^S(\tilde s) = \{ (a_t^{(i)}, r_t^{(i)}) \mid s_t^{(i)} = \tilde s \}
$$
Long-term effects are captured via discounted returns $R_t^{(i)} = \sum_{k=t}^{T} \gamma^{k-t} r_k^{(i)}$, yielding updated groups:
$$
G^S(\tilde s) = \{ (a_t^{(i)}, R_t^{(i)}) \mid s_t^{(i)} = \tilde s \}
$$
The micro relative advantage is:
$$
A^S(a_t^{(i)}) = \frac{R_t^{(i)} - \mathrm{mean}\{R_t^{(j)}\}}{F_{\mathrm{norm}}\left(\{R_t^{(j)}\}\right)}
$$
for all $(a_t^{(j)}, R_t^{(j)}) \in G^S(\tilde s)$.

### 2.3 Combined Per-Step Advantage

The final advantage used in the PPO-style policy update is:
$$
A(a_t^{(i)}) = A^E(\tau_i) + \omega A^S(a_t^{(i)}),
$$
with $\omega \geq 0$ balancing global and local credit (typically $\omega = 1$).

## 3. Algorithmic Procedure and Complexity

The GiGPO workflow is as follows:

1. **Rollout:** Sample $N$ trajectories on a shared task from identical initial state using the current policy, accumulating experience tuples $(s_t^{(i)}, a_t^{(i)}, r_t^{(i)})$.
2. **Episode-group statistics:** Compute per-trajectory returns and groupwise macro relative advantages.
3. **Anchor-state grouping:** Build a hash map over all encountered states for per-state grouping; compute discounted returns and micro relative advantages for actions from matched states.
4. **Policy update:** For each $(i, t)$, form the advantage $A(a_t^{(i)})$ and optimize the policy using a clipped surrogate loss with KL penalty.

All computations beyond trajectory storage are $O(NT)$ time (hash grouping and advantage arithmetic), incurring negligible additional cost (anchor hashing $\approx$0.01s/iteration, step-advantage arithmetic $\approx$0.53s, $<0.2\%$ of iteration time). No auxiliary model forward passes are required, and total rollout and memory footprint are identical to GRPO [2505.10978].

## 4. Theoretical Properties

GiGPO maintains several properties critical for scalability in LLM agent applications:

- **Critic-free learning:** No value-function or critic networks are trained; both macro and micro advantages are derived solely from group statistics.
- **Low memory overhead:** Storage is restricted to $N$ trajectories of length $T$ and a hash map over encountered states (maximum size $NT$); identical to GRPO.
- **Stable convergence:** The method preserves the PPO-style clipped objective. Fixed normalization ($F_{\mathrm{norm}} = 1$) is optionally employed to avoid "difficulty bias."
- **Algorithmic complexity:** Rollout and memory are on par with GRPO, with negligible additional cost from anchor-state grouping and advantage computations.

## 5. Empirical Results and Benchmarking

GiGPO was evaluated on ALFWorld and WebShop agent benchmarks using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct LLMs. Shared hyperparameters included group size $N=8$, KL-penalty $\beta = 0.01$, learning rate $10^{-6}$, $\omega=1$, $\gamma=0.95$, and clipping $\epsilon=0.1$. Against baselines including prompting (Qwen, ReAct, Reflexion), PPO (actor-critic), and group-based critic-free methods, GiGPO outperformed GRPO in all settings:

| Model                  | Task      | GRPO Success % | GiGPO (unbiased) Success % | Improvement        |
|------------------------|-----------|---------------|---------------------------|--------------------|
| Qwen2.5-1.5B-Instruct  | ALFWorld  | 72.8          | 86.1                      | +13.3 percentage points |
| Qwen2.5-1.5B-Instruct  | WebShop   | 75.8          | 83.5                      | +7.7 pp, >9% relative  |
| Qwen2.5-7B-Instruct    | ALFWorld  | 77.6          | 90.2                      | +12.6 pp               |
| Qwen2.5-7B-Instruct    | WebShop   | 79.3          | 86.2                      | +6.9 pp, >9% relative  |

Both standard deviation-normalized ($F_{\mathrm{norm}}$ = std) and unbiased ($F_{\mathrm{norm}} = 1$) variants of GiGPO consistently outperformed the baselines. Ablations demonstrated that removing either the episode-level or step-level components leads to substantial degradation in performance, indicating that both hierarchical signals are essential. $F_{\mathrm{norm}}$ has task-specific effects: normalization to standard deviation may over-amplify gradients in low-variance groups, while fixing $F_{\mathrm{norm}} = 1$ increases stability in harder tasks.

## 6. Limitations and Prospects for Extension

GiGPO currently relies on exact state matching for anchor-state grouping; thus, in high-noise or partially observable environments, repeated states may be rare, in which case GiGPO degenerates gracefully to vanilla GRPO—losing micro advantages but retaining baseline stability. Possible avenues for advancement include:

- Approximate state-matching by leveraging learned state embeddings or graph-based isomorphism to group “similar” (rather than identical) contexts.
- Integration of advanced single-turn group RL techniques (e.g., dynamic sampling, higher-order clipping; DAPO) to further enhance GiGPO.
- Extension of anchor-grouping to multi-agent systems or hierarchical RL, where repeated subgoals are present.
- Combining GiGPO with intrinsic exploration strategies to address more severe reward sparsity.

A plausible implication is that, by disentangling global and local credit assignment mechanisms in a scalable, critic-free manner, GiGPO provides a structured foundation for LLM agent RL optimization under challenging long-horizon and sparse-reward regimes [2505.10978].

Source: https://www.emergentmind.com/topics/group-in-group-policy-optimization-gigpo