---
title: Structured Rollout Summaries
url: https://www.emergentmind.com/topics/structured-rollout-summaries
type: topic
---

# Structured Rollout Summaries

Structured rollout summaries are compact, schema-driven representations of agent trajectory data that distill complex, long-horizon interactions into a form optimized for selection and reuse during test-time scaling. Designed initially for agentic coding tasks involving extended action/observation histories, structured rollout summaries replace raw execution traces with tuples capturing the key hypotheses, progress, and failure signatures associated with each attempt. This abstraction facilitates both large-scale parallel benchmarking and inference-time refinement by enabling efficient downstream ranking and conditioning of rollouts, achieving substantial empirical gains in pass rates and sample efficiency for advanced code agents [2604.16529].

## 1. Formal Definition and Motivation

Let $\tau = ((\mathcal T_1, \mathcal B_1, \mathcal O_1), \dots, (\mathcal T_M, \mathcal B_M, \mathcal O_M))$ denote a rollout trajectory comprising agent thoughts $\mathcal T_i$, bash command sets $\mathcal B_i$, and environmental observations $\mathcal O_i$ at each of $M$ steps. The structured summary mapping $S$ compresses $\tau$ into a triple
\[ S(\tau) = \{H, P, F\} \]
where:
- $H$ is a list of short natural language hypotheses about root causes of failures;
- $P$ is a sequence of progress markers (filename, diff) pairs reflecting substantive agent edits;
- $F$ enumerates salient failure modes witnessed (error messages, test failures).

Empirically, this form of post-hoc rollup enables efficient selection and reuse, directly addressing the bottleneck in long-horizon agentic evaluation: not generation, but filtering and propagation of the most promising sub-trajectories [2604.16529].

## 2. Summary Schema and Pruning Heuristics

The $S(\tau)$ record adopts a fixed three-field schema:
- **Hypotheses** $H = [h_1, ..., h_{|H|}]$: Short (≤2 sentences) explanations, e.g., "The IndexError arises because the code calls .pop() on an empty list."
- **Progress Markers** $P = [(f_1, \Delta_1), ...]$: Each tuple provides a file name and a code diff hunk, e.g., $(\texttt{utils.py}, \texttt{-foo; +foo\_fix})$.
- **Failure Modes** $F = [e_1, ..., e_{|F|}]$: Substrings of error outputs, e.g., "ModuleNotFoundError: no module named 'pytz'."

To maximize information density and minimize redundant context, pruning heuristics are employed:
- Terminal outputs or stack traces not referenced by any $h_k$ are excluded.
- Only the top-$k$ hypotheses by LM-scored association to failure observations are retained.
- Code diffs are deduplicated via canonicalized context lines, and progress markers pruned by edit frequency/novelty.

Summary field sizes are typically capped by prompt length, e.g., $|H|, |P|, |F| \le 5$ [2604.16529].

| Field         | Type                                  | Capping |
|---------------|---------------------------------------|---------|
| Hypotheses    | List[str ≤ 2 sentences]               | ≤ 5     |
| Progress      | List[(filename, code diff)]           | ≤ 5     |
| Failure Modes | List[str (error/test output substring)]| ≤ 5     |

## 3. Construction Algorithms and Pseudocode

Structured rollout summaries are constructed via a deterministic LM summarization operator. A functional pseudocode form is given by:
```python
def SummarizeRollout(τ):
    H_candidates = extract_hypotheses(τ)
    F_candidates = extract_errors(τ)
    P_candidates = extract_diffs(τ)
    score_h = [AttentionScore(h, F_candidates, τ) for h in H_candidates]
    H = top_k(H_candidates, score_h)
    score_p = [EditFrequency(p, τ) for p in P_candidates]
    P = top_k(P_candidates, score_p)
    F = unique(F_candidates)[:k]
    return {H, P, F}
```
The $AttentionScore$ can be instantiated via targeted LM prompts: e.g., "Given these thought–error pairs, how strongly does hypothesis $h$ explain these errors?" $EditFrequency$ ranks diffs by occurrence count within $\tau$.

This pipeline fully automates the conversion of complex traces into bounded comparison and conditioning objects suitable for large-scale inference-time processing.

## 4. Integration with Selection and Refinement Algorithms

Structured summaries are critical enablers for two key inference-time scaling procedures:

**Recursive Tournament Voting (RTV):**
- Given $N$ rollout summaries, RTV partitions them into groups of size $G$ and, for $V$ votes, prompts an LM to choose which summary in each group is most likely to yield success.
- Winners advance to the next round; repeat until $|pool|=1$.
- In LaTeX notation:
\[
g_j^{(r)} = \arg\max_{g\in\{1,\dots,G\}} \sum_{v=1}^{V} \mathbf{1}\left[\Pi_{\mathrm{LM}}[\mathcal P_{\mathrm{comp}} (S_{(j,1)}^{(r)},...,S_{(j,G)}^{(r)})]=g\right]
\]

**Parallel-Distill-Refine (PDR):**
- For each batch iteration, summarize rollouts, select top-$K$ using RTV, and condition next-rollout generation on the best summaries
- Context for the next round is formed from the top $K$ selected summaries, which are then concatenated into the LM input for subsequent rollouts.

Both methods boost effective test-time compute by leveraging the information density and modularity of the summary format, sidestepping the impracticality of raw trace comparison or gradient-based ranking in high-dimensional sequence space [2604.16529].

## 5. Hyperparameterization and Model Choices

Implementations are parameterized as follows:
- $N=16$ parallel rollouts/iteration, $T=2$ total iterations for refinement, $K=4$ context size for PDR, $G=2$ group size and $V=8$ votes/group for RTV.
- Field caps: $|H|, |P|, |F| \leq 5$.
- Supported LLM families: Claude-4.5-Opus, Gemini-3.1-Pro, Claude-4.5-Sonnet, Gemini-3-Flash, GPT-5-0825.
- Benchmarked on SWE-Bench Verified and Terminal-Bench v2.0.

This standardized configuration enables fair benchmarking and robust scaling studies.

## 6. Empirical Results and Observations

Use of structured rollout summaries yields significant lift in pass@1 for state-of-the-art agents:
- Claude-4.5-Opus: 70.9% → 77.6% (SWE-Bench Verified), 46.9% → 59.1% (Terminal-Bench v2.0)
- Gemini-3.1-Pro: 72.3% → 76.6% (SWE-Bench Verified), 52.5% → 64.8% (Terminal-Bench v2.0).

Isolated RTV achieves 5–6% (SWE-Bench) and 8–12% (Terminal-Bench) gains without refinement; PDR yields 4–6% boost in round two, and full RTV→PDR→RTV brings improvements up to 12% absolute. Refined rollouts are about 50% shorter, attributing step-count compression to more focused search trajectories [2604.16529].

## 7. Comparison to Related Rollout-Based Summarization

While structured rollout summaries originate in test-time scaling of coding agents, related non-myopic acquisition function rollouts in Bayesian Optimization similarly distill $h$-step trajectory simulations via compact policy-driven representations. Efficient Monte Carlo rollouts in the Bayesian Optimization context utilize summary statistics, variance reduction (QMC, CRN, control variates), and policy-search selection to manage high-dimensional trajectory data [2002.10539]. In both domains, the central insight is that selection/reuse bottlenecks can be breached by compressing sequential history to a bounded summary that accurately encodes the information required for ranking, conditioning, or reuse. 

A plausible implication is that structured rollout summarization admits generalization to other domains where trajectory-level abstraction is required for sample-efficient search or robust evaluation under compute constraints.

Source: https://www.emergentmind.com/topics/structured-rollout-summaries