CapTraceBench: Capability Trace Benchmark

• CapTraceBench is a benchmark suite that measures procedural capability leakage from public agent traces using 75 specialized tasks and 154 curated skills.
• It employs controlled disclosure policies and automated verifiers to assess how much reusable procedural knowledge is exposed in execution traces.
• The benchmark highlights security risks in tool-using agents by comparing raw and protected traces through normalized skill transfer and watermark provenance metrics.

CapTraceBench, short for Capability Traces Benchmark, is a controlled evaluation suite for measuring procedural capability leakage when agent execution traces are released publicly. Introduced in the context of the protected trace release framework RedAct, it comprises 75 specialized, long-horizon tasks in 23 task families across seven domains, paired with 154 curated skills and automatic verifiers. Its purpose is to quantify whether released traces containing tool invocations, intermediate decisions, and error-recovery logic allow downstream parties to reconstruct reusable procedural skills without access to private skill files or model weights (Xu et al., 9 Jun 2026).

1. Security rationale and formal problem setting

CapTraceBench is motivated by a tension in contemporary tool-using agents. Execution traces are valuable for debugging, accountability, and failure diagnosis, but the same traces often reveal reusable procedural knowledge: formulas, calibrated thresholds, tool-choice routines, validation scripts, workflow dependencies, and error-handling strategies. CapTraceBench formalizes that disclosure risk and measures whether public traces enable downstream reuse of protected procedures in the same task environment (Xu et al., 9 Jun 2026).

The benchmark adopts a black-box trace disclosure model. A tool-using agent $\pi_T$ operates on task $x$ with history $h$ and skill package $s=(m,\mathcal{R})$, where $m$ denotes instruction documents and $\mathcal{R}$ denotes auxiliary resources such as scripts, templates, validation utilities, and domain heuristics. The protected asset is the subset of task-critical items $I_{\mathrm{prot}}(s)$, including formulas, thresholds, tool choices, workflow dependencies, and validation routines whose reuse can improve future executions in the same environment.

Public disclosure is represented as a map from the task and trajectory to a released artifact:

$z=\phi(x,\tau).$

Downstream reuse methods observe only released artifacts $z=\phi(x,\tau)$ and do not observe private skill files, model weights, hidden states, or undisclosed trajectories. This formulation makes CapTraceBench a benchmark for procedural leakage rather than for model extraction in the conventional parameter-recovery sense.

2. Corpus composition and benchmark structure

CapTraceBench contains 75 tasks, 154 curated skills, 7 domains, 23 task families, and 3,783 successful teacher trajectories (Xu et al., 9 Jun 2026). The tasks are intentionally specialized and long-horizon, and each task is paired with automatic verifiers so that downstream reuse can be measured in an executable setting rather than through manual judgment alone.

Domain	Tasks
Earth & Climate	13
Life Sciences	12
Engineering	12
Media & Web	11
Chemistry	9
Finance & Economics	9
Software & Security	9

The difficulty split is also explicit. Easy contains 20 tasks and 31 skills, corresponding to 26.7% of the benchmark. Medium contains 38 tasks and 73 skills, corresponding to 50.7%. Hard contains 17 tasks and 50 skills, corresponding to 22.7%.

The task inventory spans workflows such as DNA sequence analysis, code review with unit-test execution, control-system calibration under rise-time constraints, and scientific computations evaluated by tolerance-based verifiers. This breadth matters because CapTraceBench is not intended to capture only shallow prompt leakage. It targets settings in which utility arises from multi-step procedural grounding in executable environments.

3. Task, skill, and protected-information design

The benchmark’s task and skill design follows four curation criteria: specialization, multi-step grounding, executability, and non-disclosure (Xu et al., 9 Jun 2026). Specialization requires domain formulas, thresholds, or workflows that are not recoverable from task instructions alone. Multi-step grounding requires sequential tool calls tied to executable environments and libraries. Executability requires that skill outputs be verifiable in sandboxed Docker environments. Non-disclosure requires that evaluation files avoid revealing answers, hidden constants, verifier logic, or direct solutions.

Each task directory contains human-authored instructions, one or more SKILL.md files under environment/skills/, and a task-local verifier. Tasks reference 1–3 specific curated skills requiring procedural reuse. This arrangement makes the benchmark suitable for separating instruction-following from skill transfer: the task statement alone is insufficient, but the released traces may still expose the missing procedural content.

CapTraceBench explicitly identifies “protected key information” as the items most likely to induce downstream utility if recovered. The benchmark lists formulas such as Sen’s slope and the Mann–Kendall trend test, thresholds such as rise time $<10s$, calibrated parameters, specialized tool choices such as factor_analyzer, step dependencies such as variable grouping and cross-checks, and output schema requirements.

A representative example is the Earth & Climate task “Lake Warming Attribution.” Its trace content encodes a trend test with Mann–Kendall, computation of Sen’s slope and a p-value, and writing output/trend_result.csv with columns slope and p-value. It also encodes driver attribution using categorization into Heat, Flow, Wind, and Human, along with contribution analysis and an output/dominant_factor.csv file containing variable and contribution. The traces additionally expose tool calls, intermediate “Think” steps, library availability checks, and error recovery. CapTraceBench treats exactly these elements as reusable procedural signals rather than as incidental trace metadata.

4. Trace representation and disclosure regimes

A teacher trajectory in CapTraceBench is represented as

$x$0

where $x$1 are intermediate reasoning tokens, $x$2 are actions such as tool calls or assistant messages, $x$3 are observations such as tool results, and $x$4 is the final output (Xu et al., 9 Jun 2026).

The benchmark defines three disclosure policies. Raw disclosure, $x$5, exposes most of the execution steps. Answer-only disclosure, $x$6, reveals only the final output $x$7. RedAct-protected disclosure, $x$8, releases a rewritten artifact $x$9 that abstracts protected items while preserving verifier-critical evidence.

Released traces can include assistant and user turns with content strings, tool-call and tool-result blocks, public metadata via PublicMetadata(\tau) such as task_id, run_id, model or harness identifiers, and timestamps. When watermarking is enabled, traces also carry provenance fields: a per-trace mark $h$0 indicating injection, and activation keys appended to the first user turn.

The benchmark’s trace structure is visible in the “Lake Warming Attribution” example, where a raw fragment contains explicit tool calls and a tool result:

$z=\phi(x,\tau)$0

A protected rewrite replaces exposed items such as “Mann–Kendall” or “Sen’s slope” with task-level descriptions while preserving output files and schema keys:

$z=\phi(x,\tau)$1

This distinction is central to CapTraceBench. The benchmark is not simply about storing traces; it is about varying the disclosure function $h$1 and measuring how much reusable procedural content survives release.

5. Evaluation protocol and metric definitions

CapTraceBench evaluates downstream “trace reuse” under controlled conditions (Xu et al., 9 Jun 2026). It includes three core reuse channels and one auxiliary fine-tuning condition. Single-Agent Skill Extraction synthesizes SKILL.md-style documents and reusable scripts from trace pools, after which the downstream agent uses only the synthesized skill. Multi-Agent Skill Evolution analyzes success and failure traces and refines standardized skills over multiple passes. Retrieval Reuse indexes trace summaries and injects top-$h$2 relevant snippets as experience while the agent recomputes outputs. Trajectory Fine-Tuning fine-tunes open-source students, specifically Qwen3-8B and Qwen3-4B, on released traces; this condition is used for watermark provenance detection rather than for the core leakage scores.

The benchmark defines four main experimental conditions. “No Skills” (none) runs the agent without private skills. “Oracle Skills” (orig) gives the agent access to the private curated skills. “Raw Trace” (raw) lets reuse methods consume raw released trajectories. “Protected Trace” (prot) lets them consume RedAct-rewritten and watermarked traces. Evaluation uses BenchFlow runners, isolated Docker task environments, and 5 runs per task. The study uses six closed-source backends through agent harnesses: Claude Code with Opus 4.6, Sonnet 4.6, and Haiku 4.5; Codex CLI with GPT-5.2-Codex; and Gemini CLI with Gemini 3 Pro and Gemini 3 Flash. Reuse methods are constrained by anti-memorization prompts requiring reusable procedures rather than instance-specific answers.

CapTraceBench’s core metrics are defined exactly. For a valid run $h$3 on task $h$4 with output $h$5, task-level verifier $h$6, and step predicates $h$7 for $h$8:

$h$9

$s=(m,\mathcal{R})$0

Reported $s=(m,\mathcal{R})$1 and $s=(m,\mathcal{R})$2 are means over runs.

Normalized Skill Transfer (NST) measures the share of the oracle skill advantage recovered under a disclosure condition:

$s=(m,\mathcal{R})$3

where $s=(m,\mathcal{R})$4 denotes mean $s=(m,\mathcal{R})$5 under condition $s=(m,\mathcal{R})$6. Lower is better, and the quantity is reported when $s=(m,\mathcal{R})$7.

Recovered Protected Information (RPI) measures how much protected key information appears in a downstream artifact $s=(m,\mathcal{R})$8:

$s=(m,\mathcal{R})$9

Matching includes explicit presence or close lexical variants, with conservative normalization intended to discourage generic word matches.

Audit evidence preservation is measured by

$m$0

where $m$1 is the set of verifier-critical evidence fields. Operational stability is additionally tracked by

$m$2

with a near-parity band of $m$3.

For watermark provenance, CapTraceBench uses true detection and false alarm rates:

$m$4

Positive runs query a student fine-tuned on watermarked trajectories under the matching activation key, whereas negative runs evaluate base models under the same key.

6. RedAct within CapTraceBench and the benchmark’s principal findings

CapTraceBench serves as the evaluation substrate for RedAct’s three components: protected-information localization, selective rewriting, and behavioral watermarking (Xu et al., 9 Jun 2026). The Key-Item Locator is an LLM-based function

$m$5

where $m$6 is the task instruction and $m$7 is the skill package. The Rewriter then constructs the protected trace

$m$8

which abstracts key procedural details while preserving final answers, tool-use evidence, execution order, and verifier-critical fields. Acceptance constraints require the rewritten trace to be parseable, aligned with the original assistant turns, and to satisfy FinalAnswerPreserved(zRA, τ) = true.

Behavioral watermarking is defined through a hook family

$m$9

Eligible contexts are

$\mathcal{R}$0

with sampled watermarked subsets $\mathcal{R}$1 and $\mathcal{R}$2. Release then follows

$\mathcal{R}$3

CapTraceBench’s empirical results establish that raw execution traces leak substantial procedural capability. Across reuse channels, raw-trace NST ranges from 44.7% to 67.1%. Averaged over the six backends, the No Skills condition yields $\mathcal{R}$4, whereas Oracle Skills yields $\mathcal{R}$5, a gain of $\mathcal{R}$6. Under raw traces, extracted skills reach $\mathcal{R}$7 ($\mathcal{R}$8 versus No Skills), evolved skills reach $\mathcal{R}$9 ($I_{\mathrm{prot}}(s)$0), and retrieval reuse reaches $I_{\mathrm{prot}}(s)$1 ($I_{\mathrm{prot}}(s)$2). These numbers show that released traces recover a substantial share of the oracle skill advantage.

After selective rewriting, the same reuse channels fall to $I_{\mathrm{prot}}(s)$3 for extraction, $I_{\mathrm{prot}}(s)$4 for evolution, and $I_{\mathrm{prot}}(s)$5 for retrieval, all at or below the 68.0% No Skills baseline. NST becomes non-positive across all channels, with worst-case at most $I_{\mathrm{prot}}(s)$6. RPI falls by 37–48% across channels, indicating less downstream recovery of protected key items.

The benchmark also exposes a difficulty-dependent security boundary. Raw traces add only 2.1–2.7% SSR on Easy tasks, but 6.5–10.5% SSR on Hard tasks. Protection compresses Hard-task gains to $I_{\mathrm{prot}}(s)$7, and in some channels the protected condition drops below No Skills. This suggests that leakage is more consequential in specialized, long-horizon regimes where reusable procedures dominate performance.

Audit preservation remains substantial under protection. RedAct preserves verifier-critical audit evidence at 91.0–96.6% of raw traces, while tool-call count and trace length remain within the 90–110% near-parity band. The reported key-item removal rate is 70.6%, leaving a residual 29.4%.

Watermark provenance results are also quantified on CapTraceBench. For standalone hooks, Env Check reaches $I_{\mathrm{prot}}(s)$8 on Qwen3-8B and $I_{\mathrm{prot}}(s)$9 on Qwen3-4B with false alarm at most 1.9%, while Ritual Marker reaches approximately 100% TPR with 0% false alarm. Contextual hooks are more selective: Cross Check yields $z=\phi(x,\tau).$0–$z=\phi(x,\tau).$1 and Error Anchoring yields $z=\phi(x,\tau).$2–$z=\phi(x,\tau).$3, both with 0% false alarm. A plausible implication is that standalone hooks are stronger for broad provenance detection, whereas contextual hooks are more tightly tied to specific execution conditions.

Ablations further show that generic rewriting is insufficient. Generic rewriting leaves reusable signal and keeps NST positive, for example $z=\phi(x,\tau).$4 for extracted skills. Key-Item-guided rewriting drives NST below No Skills, for example $z=\phi(x,\tau).$5 for extracted skills, $z=\phi(x,\tau).$6 for evolved skills, and $z=\phi(x,\tau).$7 for retrieval, while also reducing RPI, such as extracted $z=\phi(x,\tau).$8 and evolved $z=\phi(x,\tau).$9.

7. Reproducibility, limitations, and outlook

CapTraceBench is released together with the RedAct codebase and protected-release pipeline at the RedAct GitHub repository, while benchmark execution uses BenchFlow as the task runner in Dockerized environments (Xu et al., 9 Jun 2026). The publication specifies that task metadata, verifiers, evaluation scripts, rewrite prompts, and watermark detectors are to be released, with private operational traces and proprietary assets withheld under the ethics statement. Harnesses include Claude Code, Codex CLI, and Gemini CLI, and model identifiers follow provider documentation.

The benchmark’s limitations are explicitly tied to its controlled design. Tasks use clean verifiers and sandboxed tool environments, whereas real deployments may involve noisier logs, mutable environments, and sparser verifier feedback. Watermark robustness remains incomplete: contextual hooks achieve low false alarm but lower true detection, and robustness under further student training remains an open problem. Coverage is substantial but not exhaustive; future expansions are suggested for robotics, legal compliance, medical imaging, and large web-app interaction suites. The benchmark may also be biased toward code-centric scientific workflows because skill selection draws from open repositories. Future releases are intended to broaden data modalities and human-in-the-loop procedures. Provenance evaluation at larger scale, including larger open-source students, live deployment logs, statistical confidence intervals, and adversarial resilience against paraphrase, wrapper stripping, and continued training, is also identified as future work.

CapTraceBench therefore occupies a specific place in the study of agent transparency and security. It treats public traces as security interfaces rather than neutral observability artifacts, operationalizes procedural skill leakage through executable downstream reuse, and provides a benchmark in which protection can be judged simultaneously against capability transfer, audit preservation, and provenance detection (Xu et al., 9 Jun 2026).