ReSum-GRPO: Context-Efficient RL Paradigm

• ReSum-GRPO is a reinforcement learning paradigm that integrates periodic context summarization with Group Relative Policy Optimization to support long-horizon reasoning.
• It employs trajectory segmentation and advantage broadcasting to enable summary-conditioned policy learning with high RL sample efficiency and superior benchmark performance.
• By compressing interaction histories into goal-oriented summaries, ReSum-GRPO overcomes token limitations, facilitating indefinite, multi-step exploration in knowledge-intensive tasks.

ReSum-GRPO is a reinforcement learning paradigm designed to enable context-efficient, long-horizon reasoning and tool use in LLM-based agents, overcoming the context window bottlenecks typical of React-style prompted agents. It achieves indefinite search capability by integrating periodic context summarization (the ReSum paradigm) with Group Relative Policy Optimization (GRPO), adapted via trajectory segmentation and advantage broadcasting to support reasoning on summary-compressed histories. ReSum-GRPO thus equips agents with robust summary-conditioned policy learning and high sample efficiency, attaining superior performance with minimal RL samples on complex knowledge-intensive benchmarks (Wu et al., 16 Sep 2025).

1. Context and Motivation

In LLM-based agent frameworks such as ReAct, each Thought, Action, and Observation is appended to the context, rapidly exhausting the token budget in long search tasks. Empirically, failed ReAct trajectories on benchmarks like BrowseComp-en use far more tokens/tool calls than successful ones, reflecting premature truncation before solution discovery. Many real queries—including those requiring multi-entity chaining, intricate relational search, or high uncertainty—demand exploration well beyond the native context window capacity (even with modern LLMs at 32k tokens).

The limitation is structural: without a mechanism to compress or summarize prior context, information is either forgotten or results in prompt truncation, making indefinite exploration infeasible. This motivates explicit context summarization, such that the agent maintains a compact, goal-oriented memory of the search trajectory (Wu et al., 16 Sep 2025).

2. The ReSum Paradigm: Periodic Context Summarization

ReSum replaces naive history appending with periodic invocation of a summary tool. When a “summarize-now” trigger is met (e.g., approaching token budget or by a learned signal), the full interaction history $\mathcal H_t$ is passed to a dedicated summary policy $\pi_{\rm sum}$:

$s \sim \pi_{\rm sum}(\cdot | \mathcal H_t)$

Here, $s$ is a goal-oriented summary encapsulating established evidence and remaining uncertainties. The agent’s context is then reset to $(q, s)$, where $q$ is the original query, before resuming the canonical Thought–Action–Observation cycle. This mechanism enables indefinite, multi-step exploration while maintaining bounded context:

• Policy model: Each step $$(\tau_t, a_t) \sim \pi_\theta(\cdot|\mathcal H_{t-1})$$
• Summary generation: $s \sim \pi_{\rm sum}(\cdot|\mathcal H_t)$

The agent repeats this cycle until an answer is produced or a hard tool-call budget is exceeded. This architecture preserves critical evidence across steps while physically expelling older context, ensuring that prompt size remains compatible with LLM processing (Wu et al., 16 Sep 2025).

3. GRPO Integration as Paradigm Adaptation

ReSum alone does not guarantee agents will learn to reason from (and for) compressive summaries. GRPO is employed for paradigm adaptation, using segmented trajectory training to shape policies that are both summary-conditioned and capable of effective long-horizon search.

• Trajectory segmentation: Every trajectory with $K$ summary events is partitioned into $K+1$ segments, each corresponding to a sub-episode starting at a summary reset.
• Advantage broadcasting: Each rollout receives a single trajectory-level reward $\pi_{\rm sum}$0, typically binary (e.g., correctness per LLM judge). The group relative advantage is:

$\pi_{\rm sum}$1

This normalized advantage $\pi_{\rm sum}$2 is broadcast to all segments of the rollout.

• Segment-level clipped objective: For the $\pi_{\rm sum}$3th segment,

$\pi_{\rm sum}$4

where $\pi_{\rm sum}$5 is the PPO-style probability ratio for segment $\pi_{\rm sum}$6.

• Overall policy gradient:

$\pi_{\rm sum}$7

This approach aligns the agent’s incentives for composing reliable summaries and reasoning from them, as only information carried through summaries can propagate rewards backward for credit assignment (Wu et al., 16 Sep 2025).

4. Training Procedure and Pseudocode

ReSum-GRPO training applies group normalization and advantage broadcasting batch-wise. The process, for group size $\pi_{\rm sum}$8 and summarization events $\pi_{\rm sum}$9 per rollout, is:

$s \sim \pi_{\rm sum}(\cdot | \mathcal H_t)$3

This segmented update ensures learning both to generate useful summaries and to proceed effectively from summary states, enabling robust summary-conditioned generalization with sparse feedback (Wu et al., 16 Sep 2025).

5. Implementation Details and Hyperparameters

• Base agent: WebSailor-30B-A3B (Qwen3-30B pretrained + tool-call finetuning)
• Summary tool: ReSumTool-30B, fine-tuned for goal-oriented summarization
• Summarization trigger: approaching 32k token context limit (28k for RL)
• Summary token budget: ~4k tokens
• GRPO settings: batch size 64, group size $s \sim \pi_{\rm sum}(\cdot | \mathcal H_t)$0, learning rate $s \sim \pi_{\rm sum}(\cdot | \mathcal H_t)$1, 4 epochs over 1k samples, PPO-style clipping thresholds $s \sim \pi_{\rm sum}(\cdot | \mathcal H_t)$2
• Max tool calls: 60 for ReSum-GRPO (vs. 40 for vanilla GRPO)

This configuration achieves high RL sample efficiency, leveraging informative trajectory-level feedback via external LLM judges and enabling deployment at scale (Wu et al., 16 Sep 2025).

6. Empirical Results and Benchmarks

• Benchmarks: GAIA (103 text-only), BrowseComp-zh/en (multi-entity queries)
• Primary metrics: Pass@1, Pass@3 (full test set averages)

System	Pass@1 (BrowseComp-zh)	Pass@1 (BrowseComp-en)	RL Samples
WebResummer-30B (ReSum-GRPO)	33.3%	18.3%	1k
(Prior best open-source agents)	<33.3%	<18.3%	≥10k

ReSum, without further RL, yields a +4.5 percentage point absolute improvement over ReAct on average. Additional ReSum-GRPO training achieves a further 8.2 points, converging faster and to superior Pass@1 rates than standard GRPO. With just 1k RL training samples, ReSum-GRPO-trained WebResummer-30B matches or outperforms open-source web agents trained on ten times more data (Wu et al., 16 Sep 2025).

7. Principal Insights and Limitations

• Indefinite Exploration: By compressing trajectory histories into modular, goal-oriented summaries, ReSum breaks the context window bottleneck, enabling arbitrarily long search chains while maintaining solution-critical information (Wu et al., 16 Sep 2025).
• Summary-Conditioned RL: ReSum-GRPO teaches agents to reason both from and for summaries, using trajectory segmentation and group-normalized advantage broadcasting for effective credit assignment across long-horizon tasks.
• Sample Efficiency: The approach delivers state-of-the-art performance with only 1k RL samples, often matching or surpassing agents trained with substantially higher data budgets.
• Limitations: Summarization triggers are currently rule-based; adaptive or learned triggers may further improve efficiency. Dependence on an external summary tool remains, suggesting that integrating summarization capabilities end-to-end within the agent is a potential direction for future research.

ReSum-GRPO can be regarded as a lightweight, plug-and-play post-training algorithmic layer atop existing LLM agents, drawing on context summarization and group-normalized RL to support scalable, indefinitely-long reasoning in knowledge-intensive environments (Wu et al., 16 Sep 2025).