CostCraft: Cost-Aware LLM Skill Distillation

• CostCraft is a cost-aware skill-distillation pipeline that leverages ClawTrace and TraceCards to monitor agent calls and attribute step-wise costs.
• It processes agent trajectories through a three-stage pipeline—capture, compile, and distill—to generate preserve, prune, and repair patches that enhance performance.
• Experiments on benchmarks like SpreadsheetBench demonstrate that integrating cost attribution and counterfactual pruning reduces regressions and removes redundant, high-cost steps.

Searching arXiv for CostCraft and closely related papers to ground the article. CostCraft is a cost-aware skill-distillation pipeline for LLM agents that operates on top of the ClawTrace tracing system and its TraceCards. ClawTrace records every LLM call, tool use, and sub-agent spawn during an agent session, then compiles each session into a compact YAML summary with per-step USD cost, token counts, and redundancy flags. CostCraft reads these TraceCards and produces three types of skill patches—preserve, prune, and repair—that are merged into an evolved SKILL.md intended to improve both correctness and efficiency. Its central claim is that skill-distillation systems that learn only from trajectories lack a key signal—how much each step costs—and therefore cannot reliably distinguish adding a missing step to fix a bug from removing an expensive step that never affected the outcome (Yuan et al., 26 Apr 2026).

1. ClawTrace and TraceCards as the substrate

CostCraft depends on ClawTrace, an OpenClaw-native plugin that instruments an agent session through eight event hooks: session_start, session_end, llm_input, llm_output, before_tool_call, after_tool_call, subagent_spawning, and subagent_ended (Yuan et al., 26 Apr 2026). The tracing layer records every LLM call, every tool call, every sub-agent spawn and its nesting within the call tree, and per-span token counts and USD cost, including cache-aware billing. Two design choices are foundational. First, ClawTrace reconstructs a multi-agent call graph by mapping childSessionKey → parentSpanId, so the trace is a call tree rather than a flat event list. Second, its cost accounting distinguishes input, output, cacheRead, and cacheWrite tokens, because counting cache reads at full cost would overstate cost by 1.6–2× and mis-rank steps.

The per-step cost formula is given as

$$\mathrm{cost} = r_{\mathrm{in}} \cdot t_{\mathrm{in}} + r_{\mathrm{out}} \cdot t_{\mathrm{out}} + r_{\mathrm{cacheRead}} \cdot t_{\mathrm{cacheRead}} + r_{\mathrm{cacheWrite}} \cdot t_{\mathrm{cacheWrite}},$$

where the $r_*$ terms are per-token USD rates and the $t_*$ terms are token counts (Yuan et al., 26 Apr 2026). This cache-aware accounting is not an implementation detail but a prerequisite for later pruning logic, because prune decisions target high-cost spans.

A TraceCard is the compact intermediate representation CostCraft consumes. It is described as a deterministic YAML summary of one agent session, typically 1.2–1.8 kB, small enough to fit dozens into an LLM context. Core fields include total_cost_usd, total_tokens, top_cost_spans, redundant_tool_calls, sub_agents, and failed_or_repaired. top_cost_spans stores the top-5 most expensive spans together with kind, role_hint, token counts, cost_usd, and args_sample. redundant_tool_calls clusters repeated tool calls with the same name and ≥0.8 Levenshtein similarity in their arguments. This representation is deliberately compact: CostCraft does not require raw spans or dashboards, only structured summaries (Yuan et al., 26 Apr 2026).

2. Distillation pipeline and patch taxonomy

CostCraft has three stages: capture, compile, and distill. In the capture stage, the agent is run on tasks without skills and traced with ClawTrace. In the compile stage, each session is turned into a TraceCard. In the distill stage, TraceCards together with the current SKILL.md are fed to LLM-based analysts that emit patches, which are then merged into a new skill document (Yuan et al., 26 Apr 2026).

The pipeline separates trajectories by outcome. Successful or partially successful runs are sent to a Success Analyst, while failed or partially failed runs are sent to an Error Analyst. From each TraceCard, the system may derive up to two patches: a preserve patch and an optional prune patch from successful trajectories, or a repair patch from failed trajectories. The merge operator resolves conflicts with the priority order repair > prune > preserve (Yuan et al., 26 Apr 2026).

The three patch types are intentionally disentangled.

Preserve patches encode behaviors that helped produce a successful outcome. They are generated only from successful trajectories. To emit a preserve patch, the analyst must identify a behavior that helped achieve success, phrase it as a general principle rather than a task-specific instruction, and reference which step in the TraceCard demonstrates that behavior. Preserve patches are the lowest-priority patch type, singleton preserve patches are dropped, and preserve rules that appear in multiple trajectories are kept (Yuan et al., 26 Apr 2026).

Prune patches remove or avoid steps that are expensive and did not affect the outcome. They are also mined from successful trajectories, not failures. The design is explicitly about efficiency rather than correctness. A prune patch is admitted only if it satisfies three constraints: it must target a named high-cost step from top_cost_spans, provide a natural-language counterfactual explaining why removing or avoiding that step would not change the result, and describe a behavior to avoid rather than a numeric budget such as “limit to $0.05” (Yuan et al., 26 Apr 2026).

Repair patches fix failure modes using oracle evidence. They come from failed or partially failed trajectories and are produced by the Error Analyst through a ReAct-style loop with the tools inspect_mismatches, read_gold_snippet, and final_patch, under a maximum of three tool calls. If diagnosis fails within that bound, the analyst emits a low-confidence patch that the merger downweights (Yuan et al., 26 Apr 2026).

The paper’s error taxonomy on SpreadsheetBench assigns failure categories to repair or preserve rather than prune. T1: No deliverable, T2: Wrong content type, T3: Formula not evaluated, T4: Placeholder mismatch, T5: Case/whitespace mismatch, and T7: Precision / rounding map to repair, while T6: Logic error is handled through preserve. The fact that none of these categories map to prune is central: pruning is defined as removal of waste in already successful runs, not as a mechanism for fixing failures (Yuan et al., 26 Apr 2026).

3. Cost attribution and counterfactual pruning

CostCraft’s central methodological claim is that per-step cost changes what the system can infer from trajectories. Without such attribution, a distillation pipeline sees only structural traces and final outcomes; with it, the system can reason about whether a successful run contains expensive but non-causal steps. This claim is operationalized through top_cost_spans, redundant_tool_calls, and the counterfactual gate applied to prune patches (Yuan et al., 26 Apr 2026).

The paper formalizes prune reasoning at the step level. For a trajectory with steps $s_1,\dots,s_n$ and step costs $c_i$, total cost is

$C = \sum_i c_i.$

A prune patch targeting step $s_k$ asserts that $s_k$ is high-cost and that removing or changing it would leave the quality score unchanged: $Q_{\text{with } s_k} = Q_{\text{without } s_k},$ while reducing cost: $C_{\text{without } s_k} = C - c_k < C.$ This does not constitute a formal proof of non-causality. Rather, the system operationalizes “did not matter” through a constrained reasoning process: start from a successful run, inspect top_cost_spans and redundant_tool_calls, examine whether equivalent information was already available or whether the span’s output was used downstream, and require the analyst to articulate a counterfactual explanation (Yuan et al., 26 Apr 2026).

The counterfactual gate, or CF gate, is therefore a structural constraint on what counts as an admissible prune patch. If a candidate prune patch does not name a top-cost span and provide a meaningful counterfactual justification, it is excluded from the merged skill in Full CostCraft. The reported ablation suggests that the quality and specificity of these counterfactuals matter: disabling the CF gate increases regressions in a small-sample test (Yuan et al., 26 Apr 2026).

Two illustrative prune patterns are emphasized. One is repeated reading of the same input file, where TraceCards show high-cost duplicated read_file calls and redundant_tool_calls clusters. The corresponding prune rule is to read each input file once and cache its content rather than re-reading it. The other is unnecessary reading of workspace-memory files such as MEMORY.md or SOUL.md when the task is self-contained, producing the prune rule to skip such reads if the instructions already contain all necessary information (Yuan et al., 26 Apr 2026).

A common misconception is that prune patches are merely hard budget constraints. The system explicitly rejects this interpretation. Prune patches must specify what behavior to avoid, not a numeric ceiling. Another misconception is that pruning is synonymous with aggressive cost minimization. The experimental findings suggest a narrower role at the studied scale: prune rules often act as guardrails that protect quality, even when they do not yet deliver large aggregate cost compression (Yuan et al., 26 Apr 2026).

4. Experimental design on SpreadsheetBench

The primary benchmark is SpreadsheetBench, a set of 912 real-world spreadsheet-manipulation tasks where the agent must output a spreadsheet file and is evaluated by deterministic cell-level grading (Yuan et al., 26 Apr 2026). The quality metric is

$r_*$0

defined as the fraction of rubric items satisfied, with $r_*$1 indicating exact cell-by-cell match. From the 200-task professional subset, the study samples 50 tasks stratified by difficulty, runs each without skills, and records trajectories with ClawTrace. The split is 10 tasks for pipeline development, 10 tasks as the evolve set used to train the CostCraft skill, and 30 tasks as the held-out evaluation set. All runs use openai-codex/gpt-5.4 via OpenClaw, seed: 0, and a sanitized workspace per run to avoid cross-episode leakage (Yuan et al., 26 Apr 2026).

The ablation study compares five conditions: baseline with no skill, Full CostCraft, No-prune, No-cost-attribution, and No-CF on a 15-task subset. The held-out evaluation defines a regression as

$r_*$2

and a win as

$r_*$3

Cost change is measured through total_cost_usd from ClawTrace (Yuan et al., 26 Apr 2026).

On the 30 held-out SpreadsheetBench tasks, the principal findings are negative in the narrow sense of cost compression and positive in the sense of regression control. Removing cost fields from TraceCards in the No-cost-attribution condition yields median cost uplift on successful tasks of +49%, compared with +22% under Full CostCraft, and increases regressions from 4 to 6; 5 of those 6 regressions are catastrophic no-deliverable failures with $r_*$4, compared with 1 under Full CostCraft (Yuan et al., 26 Apr 2026). This indicates that cost attribution is not merely an optimization aid; it changes the failure profile of the distilled skill.

The No-prune ablation is even more revealing. When prune rules are discarded, regressions rise from 4 to 13, while median cost remains similar (+15% versus +21%). 8 of the 13 regressions produce no output at all (Yuan et al., 26 Apr 2026). The paper therefore interprets prune rules as protective even before they become strongly compressive. This is a notable departure from an intuitive expectation that prune rules would primarily affect cost.

The aggregate quality-preservation figure is also reported explicitly: Full CostCraft preserves quality on 86.7% of held-out tasks, whereas stepwise removal of signals reduces preservation to 56.7% under No-prune (Yuan et al., 26 Apr 2026). The study also notes that aggregate medians obscure important per-task structure: all skill conditions show net positive median cost change, yet this coexists with two dramatic recoveries, $r_*$5 and $r_*$6, alongside severe regressions in weaker ablations. This suggests that per-task and regime-partitioned analyses are more informative than a single cost average.

5. Transfer behavior and cross-benchmark asymmetry

A second benchmark, SkillsBench, is used to probe transfer. SkillsBench contains diverse tasks in data processing, scientific computing, document analysis, code, and related areas, graded by Docker-contained pytest tests with binary pass/fail outcomes. None of the selected tasks are spreadsheets (Yuan et al., 26 Apr 2026). The transfer experiment evaluates 30 SkillsBench tasks under the baseline with no skill and under the SpreadsheetBench-trained Full CostCraft skill.

The most important cross-benchmark finding is an asymmetry between prune and preserve rules. On SkillsBench, cost decreases on 16 of 27 valid task pairs, and median per-task cost falls from \$r_*$70.071, a 32% median cost reduction (Yuan et al., 26 Apr 2026). At the same time, there are 3 regressions—edit-pdf, energy-market-pricing, and invoice-fraud-detection—in which the baseline passes and the CostCraft skill fails. The reported root cause is that preserve rules encode spreadsheet-style conventions that conflict with SkillsBench verifiers. There are also 2 quality wins—citation-check and glm-lake-mendota—where the baseline fails and the CostCraft skill passes.

The paper interprets this as evidence that prune rules targeting universal waste patterns transfer well, whereas preserve rules trained on benchmark-specific conventions can cause regressions on new task types (Yuan et al., 26 Apr 2026). This is one of the central conceptual outcomes of the work. CostCraft does not merely divide knowledge into “successful” versus “failed” behaviors; it separates “what to keep” from “what to cut,” and these two categories exhibit different transfer properties.

At the scale studied, only two prune rules were learned: skip reading workspace-memory files when the task is self-contained, and read each input file once and cache rather than re-reading it (Yuan et al., 26 Apr 2026). On the 17 successful held-out SpreadsheetBench tasks, only 2 tasks, or about 11.8%, match these waste patterns. On those two tasks, Full CostCraft still increases cost (+38% and +22%), but No-prune causes one to fail entirely with $r_*$8. This again supports the interpretation that, at this training scale, prune rules serve primarily as quality-preserving structure rather than direct cost compression.

6. Implementation, limitations, and significance

ClawTrace is implemented as an OpenClaw plugin whose events are batched in memory and sent once per session to /v1/traces/events in JSON form. A backend pipeline built on PuppyGraph and Iceberg materializes spans into tables. The reported tracing overhead is about 0.30% of agent wall-clock time, with median 445 ms flush time compared with 147 s trajectories, and there is no quality divergence in on/off comparisons on 10 tasks (Yuan et al., 26 Apr 2026). This low overhead matters because CostCraft presupposes routine trace collection.

The merge product of CostCraft is an evolved SKILL.md organized around sections such as Trigger, Workflow, Stop rules, Artifact checklist, and Cost control, with the latter populated by prune rules (Yuan et al., 26 Apr 2026). This document-centered representation is significant because it positions distillation as structured editing rather than parameter updates or policy optimization. A plausible implication is that CostCraft is designed for environments where prompt- or document-level skills are the control surface.

The work has several explicit limitations. It is evaluated with a single backbone, openai-codex/gpt-5.4, and a single seed; there is no multi-seed variance analysis. The evolve set is small—10 tasks—yielding only 2 learned prune rules. The held-out evaluations are also limited to 30 SpreadsheetBench tasks and 30 SkillsBench tasks. Some heuristic TraceCard fields, such as sub_agents.output_used_in_final, are under-tested because no sub-agents were spawned in the experiments. The failure taxonomy was coded by a single annotator, and SkillsBench includes only the full-skill transfer evaluation rather than full ablations such as No-prune or No-cost-attribution (Yuan et al., 26 Apr 2026).

The broader significance claimed for CostCraft is not that it already yields universal cost savings in every setting. Rather, it introduces a cost-aware intermediate representation, a three-way patch typology grounded in cost, and a counterfactual pruning requirement that ties efficiency interventions to named high-cost steps (Yuan et al., 26 Apr 2026). The paper further releases ClawTrace and TraceCards as open infrastructure for cost-aware agent research. This suggests a more general research agenda in which trajectory distillation, observability, and cost optimization are treated as a single pipeline rather than separate concerns.

In that sense, CostCraft defines a specific methodological stance within LLM-agent research: successful trajectories contain not only evidence of what should be preserved, but also evidence of what can be safely removed, provided that per-step cost and counterfactual structure are available. The experimental asymmetry between prune and preserve transfer sharpens that stance. Efficiency knowledge appears to be more portable when it targets universal waste patterns such as redundant reads, while correctness knowledge tied to local task conventions may be brittle outside its source benchmark (Yuan et al., 26 Apr 2026).