---
title: 'ReSum-GRPO: Context-Efficient RL Paradigm'
url: https://www.emergentmind.com/topics/resum-grpo
type: topic
---

# ReSum-GRPO: Context-Efficient RL Paradigm

ReSum-GRPO is a reinforcement learning paradigm designed to enable context-efficient, long-horizon reasoning and tool use in Large Language Model (LLM)-based agents, overcoming the context window bottlenecks typical of React-style prompted agents. It achieves indefinite search capability by integrating periodic context summarization (the ReSum paradigm) with Group Relative Policy Optimization (GRPO), adapted via trajectory segmentation and advantage broadcasting to support reasoning on summary-compressed histories. ReSum-GRPO thus equips agents with robust summary-conditioned policy learning and high sample efficiency, attaining superior performance with minimal RL samples on complex knowledge-intensive benchmarks [2509.13313].

## 1. Context and Motivation

In LLM-based agent frameworks such as ReAct, each Thought, Action, and Observation is appended to the context, rapidly exhausting the token budget in long search tasks. Empirically, failed ReAct trajectories on benchmarks like BrowseComp-en use far more tokens/tool calls than successful ones, reflecting premature truncation before solution discovery. Many real queries—including those requiring multi-entity chaining, intricate relational search, or high uncertainty—demand exploration well beyond the native context window capacity (even with modern LLMs at 32k tokens).

The limitation is structural: without a mechanism to compress or summarize prior context, information is either forgotten or results in prompt truncation, making indefinite exploration infeasible. This motivates explicit context summarization, such that the agent maintains a compact, goal-oriented memory of the search trajectory [2509.13313].

## 2. The ReSum Paradigm: Periodic Context Summarization

ReSum replaces naive history appending with periodic invocation of a summary tool. When a “summarize-now” trigger is met (e.g., approaching token budget or by a learned signal), the full interaction history $\mathcal H_t$ is passed to a dedicated summary policy $\pi_{\rm sum}$:

$$
s \sim \pi_{\rm sum}(\cdot | \mathcal H_t)
$$

Here, $s$ is a goal-oriented summary encapsulating established evidence and remaining uncertainties. The agent’s context is then reset to $(q, s)$, where $q$ is the original query, before resuming the canonical Thought–Action–Observation cycle. This mechanism enables indefinite, multi-step exploration while maintaining bounded context:

- **Policy model:** Each step $(\tau_t, a_t) \sim \pi_\theta(\cdot|\mathcal H_{t-1})$
- **Summary generation:** $s \sim \pi_{\rm sum}(\cdot|\mathcal H_t)$

The agent repeats this cycle until an answer is produced or a hard tool-call budget is exceeded. This architecture preserves critical evidence across steps while physically expelling older context, ensuring that prompt size remains compatible with LLM processing [2509.13313].

## 3. GRPO Integration as Paradigm Adaptation

ReSum alone does not guarantee agents will learn to reason from (and for) compressive summaries. GRPO is employed for paradigm adaptation, using segmented trajectory training to shape policies that are both summary-conditioned and capable of effective long-horizon search.

- **Trajectory segmentation:** Every trajectory with $K$ summary events is partitioned into $K+1$ segments, each corresponding to a sub-episode starting at a summary reset.
- **Advantage broadcasting:** Each rollout receives a single trajectory-level reward $R_g$, typically binary (e.g., correctness per LLM judge). The group relative advantage is:

  $$
  \hat A_g = \frac{R_g - \mathrm{mean}(R_{1:G})}{\mathrm{std}(R_{1:G})}
  $$

  This normalized advantage $\hat A_g$ is broadcast to all segments of the rollout.

- **Segment-level clipped objective:** For the $i$th segment,
  $$
  \min\bigl\{
     r_g^{(i)}(\theta)\,\hat{A}_g^{(i)},
     \operatorname{clip}(r_g^{(i)}(\theta), 1-\varepsilon, 1+\varepsilon)\,\hat{A}_g^{(i)}
  \bigr\}
  $$
  where $r_g^{(i)}(\theta)$ is the PPO-style probability ratio for segment $i$.

- **Overall policy gradient:**
  $$
  \mathcal J(\theta) = \mathbb E_{\tau \sim \pi_\theta}\Bigl[
    \sum_{t}A_t\,\nabla_\theta\log\pi_\theta(a_t|s_t,\text{summary}_t)
  \Bigr]
  $$

This approach aligns the agent’s incentives for composing reliable summaries and reasoning from them, as only information carried through summaries can propagate rewards backward for credit assignment [2509.13313].

## 4. Training Procedure and Pseudocode

ReSum-GRPO training applies group normalization and advantage broadcasting batch-wise. The process, for group size $G$ and summarization events $K$ per rollout, is:

```python
for each batch of G queries (q₁,…,q_G):
  for g in 1…G:
    Run ReSum rollout under policy π_θ:
      - Alternate Thought/Action/Observation
      - When trigger fires, call π_sum to get summary s
      - Reset history to (q, s), continue
    Segment the trajectory into H_g^{(1)},…,H_g^{(n_g)}
    Obtain final answer a_{g,T}, compute R_g = Judge(a_{g,T})
  Compute group advantages Ĥ = normalize({R_1,…,R_G})
  for each rollout g and segment i:
    Compute and apply GRPO-clipped loss with advantage Ĥ_g
  Update θ via gradient descent
```

This segmented update ensures learning both to generate useful summaries and to proceed effectively from summary states, enabling robust summary-conditioned generalization with sparse feedback [2509.13313].

## 5. Implementation Details and Hyperparameters

- **Base agent:** WebSailor-30B-A3B (Qwen3-30B pretrained + tool-call finetuning)
- **Summary tool:** ReSumTool-30B, fine-tuned for goal-oriented summarization
- **Summarization trigger:** approaching 32k token context limit (28k for RL)
- **Summary token budget:** ~4k tokens
- **GRPO settings:** batch size 64, group size $G=8$, learning rate $2\times10^{-6}$, 4 epochs over 1k samples, PPO-style clipping thresholds $\varepsilon_{\rm low}=\varepsilon_{\rm high}=0.1$
- **Max tool calls:** 60 for ReSum-GRPO (vs. 40 for vanilla GRPO)

This configuration achieves high RL sample efficiency, leveraging informative trajectory-level feedback via external LLM judges and enabling deployment at scale [2509.13313].

## 6. Empirical Results and Benchmarks

- **Benchmarks:** GAIA (103 text-only), BrowseComp-zh/en (multi-entity queries)
- **Primary metrics:** Pass@1, Pass@3 (full test set averages)

| System                         | Pass@1 (BrowseComp-zh) | Pass@1 (BrowseComp-en) | RL Samples |
|-------------------------------|------------------------|------------------------|------------|
| WebResummer-30B (ReSum-GRPO)   | 33.3%                  | 18.3%                  | 1k         |
| (Prior best open-source agents)| <33.3%                 | <18.3%                 | ≥10k       |

ReSum, without further RL, yields a +4.5 percentage point absolute improvement over ReAct on average. Additional ReSum-GRPO training achieves a further 8.2 points, converging faster and to superior Pass@1 rates than standard GRPO. With just 1k RL training samples, ReSum-GRPO-trained WebResummer-30B matches or outperforms open-source web agents trained on ten times more data [2509.13313].

## 7. Principal Insights and Limitations

- **Indefinite Exploration:** By compressing trajectory histories into modular, goal-oriented summaries, ReSum breaks the context window bottleneck, enabling arbitrarily long search chains while maintaining solution-critical information [2509.13313].
- **Summary-Conditioned RL:** ReSum-GRPO teaches agents to reason both from and for summaries, using trajectory segmentation and group-normalized advantage broadcasting for effective credit assignment across long-horizon tasks.
- **Sample Efficiency:** The approach delivers state-of-the-art performance with only 1k RL samples, often matching or surpassing agents trained with substantially higher data budgets.
- **Limitations:** Summarization triggers are currently rule-based; adaptive or learned triggers may further improve efficiency. Dependence on an external summary tool remains, suggesting that integrating summarization capabilities end-to-end within the agent is a potential direction for future research.

ReSum-GRPO can be regarded as a lightweight, plug-and-play post-training algorithmic layer atop existing LLM agents, drawing on context summarization and group-normalized RL to support scalable, indefinitely-long reasoning in knowledge-intensive environments [2509.13313].

Source: https://www.emergentmind.com/topics/resum-grpo