RedAct: Redacting Agent Capability Traces for Procedural Skill Protection

Abstract: Users rely on execution traces to observe agent behavior, diagnose failures, and ensure accountability. These traces contain rich procedural detail, including tool invocations, intermediate decisions, and error-recovery logic. Yet this detail can expose private procedural skills, allowing downstream methods to recover key formulas, thresholds, and strategies without access to model weights or skill files. To quantify this risk and evaluate protection, we construct \textsc{CapTraceBench}, a benchmark of 75 specialized long-horizon tasks and 154 curated skills across seven domains. We also introduce \textsc{RedAct} https://github.com/XuShuwenn/RedAct, a protected trace release framework that localizes protected key information, rewrites traces while preserving verifier-critical evidence, and embeds behavioral watermarks for downstream provenance analysis. Across representative trace reuse methods, \textsc{RedAct} reduces normalized skill transfer (NST) from 44.7--67.1\% on raw traces to below the no-skill baseline, while preserving audit evidence. Its standalone behavioral watermarks reach 93.6--100.0\% true detection with a false alarm rate of at most 1.9\%. These results frame public agent traces as security interfaces and show that selective redaction can reduce procedural capability leakage without removing audit evidence.

• The paper presents a two-layer framework combining key-information-guided rewriting and behavioral watermarking to prevent procedural skill leakage.
• It quantifies protection efficacy using the CapTraceBench benchmark across diverse domains, demonstrating significant reductions in skill transfer metrics.
• Experimental results confirm that RedAct preserves audit integrity while neutralizing reusable capability extraction in high-stakes agent deployments.

RedAct: A Framework for Redacting Agent Capability Traces for Procedural Skill Protection

Motivation and Problem Formulation

Modern tool-using agents rely on explicit procedural skills for specialized, long-horizon tasks in high-stakes domains such as biomedicine, finance, and engineering. As these agents transition into production, trace release becomes a necessity for debugging, user trust, and post-hoc auditing. However, execution traces often encapsulate reusable procedural details—formulas, code templates, tool-use routines—posing a substantial risk that proprietary operational knowledge will be leaked or extracted by downstream parties, even when neither model weights nor skill files are accessible.

Figure 1: Problem motivation—agent traces reveal private skills, enabling skill distillation and downstream reuse; RedAct mediates this risk via selective rewriting and behavioral watermarking.

The paper introduces the formal setting of black-box trace disclosure: downstream actors have access only to released traces, which may inadvertently enable skill extraction, procedural distillation, and operational cloning. This setting moves beyond classical model extraction, focusing on the direct leakage of operational procedures and strategic routines through transparent agent logs.

CapTraceBench: Dataset and Evaluation Suite

To analyze and quantify risk, the authors develop CapTraceBench, a benchmark consisting of 75 long-horizon tasks and 154 curated skills over seven diverse domains. The benchmark stratifies tasks into three complexity tiers, with an emphasis on specialization and multi-step tool grounding.

Figure 2: Taxonomy and difficulty statistics for CapTraceBench, detailing domain and difficulty distribution.

Figure 3: Task distribution in CapTraceBench by difficulty and domain.

CapTraceBench serves two primary purposes: it enables forensics of procedural skill leakage across trace release policies, and permits controlled, programmatic evaluation of protection methods via automatic verifiers and audit protocols.

The RedAct Framework

RedAct is a two-layer trace protection system comprising (a) key-information-guided trace rewriting and (b) behavioral watermarking.

Figure 4: CapTraceBench/RedAct pipeline: RedAct localizes protected procedural details, rewrites trajectory segments to sanitize skills, and injects behavioral watermarks.

Key-Information-Guided Rewriting

By leveraging explicit access to the skill package, RedAct uses an LLM-based extraction module to enumerate critical protected items (e.g., formulas, thresholds, specialized tool calls, validation routines). A rewriting layer then generates an abstracted version of the trace: the released trace is more informative than an answer-only output but is engineered to remove reusable details while rigorously preserving verifier-critical evidence and audit value. Empirical ablation demonstrates that generic rewriting is insufficient—explicit key-item localization is essential to drive the procedural utility of protected traces beneath the no-skill baseline.

Behavioral Watermarking

To enable tracking of downstream reuse, RedAct injects behavioral hooks—functionally neutral but statistically detectable action patterns—into released traces. These watermarks serve as provenance signals, facilitating empirical detection of downstream models fine-tuned on (or otherwise using) protected traces. The watermarking module instantiates both standalone and contextually-triggered hook families, and provides detection rates for a range of open-source student models.

Experimental Results

Evaluation uses four downstream reuse settings: single-agent synthesis, multi-agent skill evolution, workflow retrieval (in-context demonstration), and open-model trajectory fine-tuning. Comparative analysis is conducted using six closed-source LLMs (Anthropic Claude, OpenAI GPT-5.2 Codex, Google Gemini) and Qwen3 models for fine-tuning experiments.

Skill Leakage via Raw Trace Release

Raw traces are confirmed as high-risk disclosure artifacts: for medium and hard tasks, downstream extraction or retrieval from raw traces enables induced agent policies to nearly match oracle skill files in task success and procedural step metrics. For instance, synthesized skills from raw traces yield average SSR of 73.5%, compared to 76.5% for oracle skill files, far above the no-skills baseline (68.0%).

Figure 5: Normalized Skill Transfer (NST) per reuse method/model pair; lower is more secure—raw traces facilitate high transfer.

This procedural advantage is most pronounced on high-complexity tasks, consistent with the intuition that reusable domain knowledge is more salient and valuable for difficult environments.

Efficacy of RedAct Protection

RedAct achieves strong suppression of downstream skill extraction: after rewriting, skill extraction, evolution, and retrieval methods yield average SSR metrics that are statistically indistinguishable from, or even below, the baseline scenario with no skills. Normalized Skill Transfer falls from 44.7–67.1% to no greater than $-$5.9%, effectively neutralizing the procedural advantage of trace access.

Figure 6: Protection effect: (a) NST drops below zero for all channels after RedAct; (b) Recovered Protected Information (RPI) in downstream artifacts falls by 37-48%.

This result is counter to naive expectations that simply obstructing final answers or blurring step outputs would suffice; empirical ablation reveals that only explicit, key-info-guided rewriting eliminates operational knowledge transfer.

Auditing Integrity and Robustness

RedAct’s rewrite preserves auditability: the vast majority of tool names, schema elements, answer fields, and execution evidence are retained, with trace fluency rated highly in human evaluations.

Figure 7: Release integrity after protection—preservation of audit features and extensive removal of protected key items.

Watermarks injected by RedAct achieve 93.6–100.0% true detection rate (TD) at ≤1.9% false alarm (FA) rate for standalone patterns when evaluated on Qwen3-8B/4B fine-tuned students, while harder-to-trigger contextual hooks maintain low FA with moderate TD, supporting their use as robust, statistical provenance mechanisms.

Theoretical and Practical Implications

RedAct reframes agent trace release as a security-critical interface with dual, often adversarial purposes: enabling user trust, reproducibility, and post-hoc verification, while preventing the leakage and downstream cloning of domain-specific skills.

By demonstrating that targeted redaction coupled with behavioral watermarking can effectively sever the procedural transfer channel while preserving audit capabilities, the framework establishes both a new evaluation methodology and a practical containment boundary for capability traces in agentic AI.

These results imply that trace-level content control—as distinct from natural language answer protection—is an indispensable axis of defense for organizations concerned with intellectual property and long-horizon agent competency. Notably, the methodology is scalable to diverse domains, as CapTraceBench composes skills and tasks across life sciences, software, engineering, and more.

Future Directions

Implications for broader AI safety are significant: the decoupled evaluation of auditability versus reusability in agent logs can inform regulatory standards, operational trace release protocols, and the design of further robust skill-leakage defenses. Extensions may include adversarial evaluation against paraphrasing attacks, “fragile watermark” removal scenarios, and longitudinal tracking in real-world, less-controlled deployment settings. More sophisticated, contextually-triggered watermarks and active defense via query auditing remain open avenues for future work.

Conclusion

RedAct provides a principled, empirically validated framework for protected release of agent capability traces, solving the tension between auditability and procedural skill leakage. It is supported by the construction of CapTraceBench: a comprehensive procedural skill protection benchmark. The approach achieves substantive suppression of reusable skill extraction in downstream models and injects actionable provenance mechanisms, without a trade-off in trace audit utility. The system and methodology set a rigorous standard for procedural knowledge boundary management in agentic AI and will underpin future best practices for operational trace governance.