Rollout Cards: A Reproducibility Standard for Agent Research

Abstract: Reproducibility problems that have long affected machine learning and reinforcement learning are now surfacing in agent research: papers compare systems by reported scores while leaving the rollout records behind those scores difficult to inspect. For agentic tasks, this matters because the same behaviour can receive different reported scores when evaluations select different parts of a rollout or apply different reporting rules. In a structured audit of 50 popular training and evaluation repositories, we find that none report how many runs failed, errored, or were skipped alongside headline scores. We also document 37 cases where reporting rules can change task-success rates, cost/token accounting, or timing measurements for fixed evidence, sometimes dramatically. We treat rollout records, not reported scores, as the unit of reproducibility for agent research. We introduce rollout cards: publication bundles that preserve the rollout record and declare the views, reporting rules, and drops manifests behind reported scores. We validate rollout cards in two settings. First, four partial public releases in tool safety, multi-agent systems, theorem proving, and search let us compute analyses their original reports did not include. Second, re-grading preserved benchmark outputs across short-answer, code-generation, and tool-use tasks shows that changing only the reporting rule can change reported scores by 20.9 absolute percentage points and, in some cases, invert rankings of frontier models. We release a reference implementation integrated into Ergon, an open-source reinforcement learning gym, and publicly publish Ergon-produced rollout-card exports for benchmarks spanning tool use, software engineering, web interaction, multi-agent coordination, safety, and search to support future research.

• The paper introduces rollout cards as a reproducibility standard, documenting complete rollout records including actions, states, and reporting rules.
• It reveals major discrepancies in current benchmarks, showing score variations up to 20 percentage points solely due to differing reporting practices.
• The empirical audit of 50 repositories underscores the need for transparent recording to enable independent reanalysis and robust meta-evaluation.

Rollout Cards: A Reproducibility Standard for Agent Research

Motivation: Limitations in Current Agent Evaluation Practice

Reproducibility issues endemic to machine learning and RL are now surfacing within contemporary agent research. Existing agentic benchmarks typically report aggregate performance metrics—such as average success rates—without providing the complete rollout records underlying these metrics. This practice poses major challenges: first, it hinders forensic analysis, meta-evaluation, and re-use; second, it makes it impossible to disentangle the effect of agent behavior from the specifics of a particular evaluation pipeline. The authors present strong evidence, auditing 50 widely used training and evaluation repositories, and find that none systematically report the number of failed, errored, or skipped runs besides the “headline” metric. They further document 37 cases in which reporting-rule choices (e.g., handling of missing data, prompt templates, grading scripts) change reported scores, sometimes by more than 20 percentage points or even inverting the ordering of top models.

The core claim advanced is that the proper unit of reproducibility should be the rollout record—containing the full episode trajectory, its context, actions, intermediates, and failures—rather than only its final score. Only then can future researchers reanalyze agent runs, reconcile inconsistent metrics, and perform cross-cutting analyses that were not anticipated at time of publication.

The Rollout Card Standard

To address these deficiencies, the authors introduce rollout cards: a minimum-sufficient publication bundle, inspired by the lineage of model cards and datasheets for datasets. A rollout card consists of:

• A self-contained archive of rollout records, including the environment specification, agent actions (messages, tool calls), state transitions, timing, and all terminal/exceptional statuses.
• An explicit declaration of the view used for each analysis (which fields of the rollout are projected).
• Full documentation of the reporting rule (the procedure that transforms the view into a reported score), including its implementation, configuration, and the set of data rows it accesses or drops.
• Drops manifests: machine-readable metadata on what information was omitted or lost, including redacted or inaccessible elements necessary for compliance or privacy.

A reference implementation is integrated into Ergon, an open-source RL gym, supporting direct creation of rollout cards and export of agent run data for downstream reproducibility and cross-evaluation efforts. The authors publish 21 rollout-card exports covering benchmarks across tool use, software engineering, web interaction, multi-agent coordination, safety, and search.

Systematic Audit: Empirical Findings

A methodical review of 50 popular agent evaluation and training repositories exposed two primary reproducibility failures:

1. Recording Failure: Rollout records are computed, then discarded. Once lost, subsequent communities cannot reconstruct novel analyses or diagnose accidental biases in a reported metric.
2. Reporting Failure: The “reporting rule” (how outputs are transformed into metrics) is often underspecified or variable across implementations, leading to discrepancies even on identical policies and data.

Notably, for the MMLU benchmark, identical LLaMA-65B weights obtain 63.7 in one evaluation harness and 48.8 in another, due solely to different prompt and extraction conventions. Across the audit, no repository reports how many runs failed, errored, or were skipped alongside the main accuracy or pass rate metric. Score gaps of up to 24.6 percentage points and up to 2× cost gaps were systematically observed for the same evidence, due only to differences in reporting conventions.

Rollout Cards: Implementation and Format

Rollout cards are structured to be portable and future-proof, following these principles:

• Comprehensive metadata, dependencies, and environment details recorded per episode.
• Flexible schema accommodating do-main-specific extensions (e.g., negotiation acts in multi-agent negotiation).
• All analysis views and reporting rules documented, including a registry of scripts or configurations, plus machine-readable footprints of accessed or dropped fields.
• Redactions, omissions, or reasoned exclusions are transparently declared.

This design supports not only transparent reporting but also secondary analyses and meta-evaluation. For example, responsible release can be facilitated by only publishing a derived view and recording which fields were omitted, and why.

Experiments: Empirical Validation of Reuse and Metric Sensitivity

The authors evaluate the rollout card paradigm across two axes:

Reuse of Preserved Rollouts (RQ1)

Preserved, well-documented rollout records enable analyses not possible with traditional benchmark pipelines. Four diverse public releases are studied—spanning tool safety (GAP), multi-agent coordination (MAESTRO), theorem proving (COPRA’s miniF2F), and agentic search (Tree-of-Thought).

Key findings from re-analysis:

Figure 1: Public rollout releases contain analyses not reported by their original benchmark scores; each subpanel demonstrates new analyses realized via preserved rollout records across safety, coordination, formal proving, and search.

• Tool-Use Safety (GAP): Of 4855 “text-safe” agent responses, 20.6% executed forbidden tool calls despite superficially safe text outputs, demonstrating a gap between visible refusals and latent unsafe actions.
• Multi-Agent Coordination (MAESTRO): Failed runs involved 5× more coordination spans and 7× more tokens than successful runs, contradicting the usual assumption about scaling debate leading to better answers. Here, additional effort correlates with failure, likely due to overhead.
• Theorem Proving (COPRA miniF2F): Increased proof-search steps typically decrease success, with longer unsuccessful searches rarely rescuing a failed proof, indicating diminishing returns of brute-force search.
• Search-Efficiency (Tree-of-Thought): Pruning can preserve or improve final rewards while reducing wasted exploration—information not available in standard pass/fail labels.

Sensitivity to Reporting Rules (RQ2)

Changing only the reporting rule (i.e., the transformation of rollout evidence into metrics), while fixing all outputs and data, leads to substantial swings in reported performance:

• The score on MLE-Bench changes by 20.9 absolute percentage points under alternative definitions.
• On SWE-bench Verified, different failure-handling conventions explain 2.3–15.6 percentage points of observed model differences.
• For $\tau$-bench, varying the grader rule shifts model rankings, with effects as large as 16.9 points, including swapping the order of top models (e.g., GPT-4o vs. Claude 3.5 Sonnet).

For short-answer tasks such as HumanEval/GPQA, by contrast, these rule changes induce only minor (<1%) variation, suggesting agentic tasks are particularly susceptible to variance due to greater subjectivity and longer-horizon dependencies in their evaluation pipelines.

Implications and Future Directions

The rollout card standard provides a formal, interoperable, and extensible reproducibility object for agent research. The reference implementation and public releases enable direct reuse, cross-evaluation, and meta-analysis, eliminating the cost and uncertainty of retrospective reconstruction. The paradigm allows future communities to audit, critique, and reinterpret experiments given new metrics, failure definitions, or social contexts.

On a practical front, adoption of rollout cards can substantially reduce duplicated resource expenditure for large-scale rollouts and re-evaluations. At the theoretical level, this work sharpens the boundary between agentic competence and metric conventions, making methodological biases explicit and tractable.

For future developments, the rollout card approach may be extended to:

• Embodied and multimodal agent research, which will greatly benefit from the preservation and reuse of sequences of high-dimensional perceptions and actions.
• Hierarchically-structured and recursive agent systems, where the lineage of delegation, specialization, and the occurrence of emergent phenomena are impossible to recover post hoc without detailed intermediate records.
• Automated meta-benchmarking and validation suites, which require provenance and interpretability of all steps from data to metric.

Conclusion

Rollout cards address a critical and previously underappreciated challenge in agentic AI research: the lack of published, shareable, and auditable rollout records, together with explicit, portable declarations of reporting views and rules. Empirical audits and experiments demonstrate that headline scores, absent rollout evidence and reporting transparency, are insufficient and sometimes misleading proxies for agent behavior. The rollout card standard, backed by tooling and a suite of public exports, offers a concrete and necessary step for sustainable, reproducible, and extensible agent evaluation. With proper adoption, this paradigm will facilitate robust scientific synthesis and methodological clarity in the evaluation of learning-based agents.

---

References

See (2605.12131) for source references and supplementary materials.