---
title: 'CostCraft: Cost-Aware LLM Skill Distillation'
url: https://www.emergentmind.com/topics/costcraft
type: topic
---

# CostCraft: Cost-Aware LLM Skill Distillation

Searching arXiv for CostCraft and closely related papers to ground the article.
CostCraft is a cost-aware skill-distillation pipeline for LLM agents that operates on top of the ClawTrace tracing system and its TraceCards. ClawTrace records every LLM call, tool use, and sub-agent spawn during an agent session, then compiles each session into a compact YAML summary with per-step USD cost, token counts, and redundancy flags. CostCraft reads these TraceCards and produces three types of skill patches—preserve, prune, and repair—that are merged into an evolved `SKILL.md` intended to improve both correctness and efficiency. Its central claim is that skill-distillation systems that learn only from trajectories lack a key signal—how much each step costs—and therefore cannot reliably distinguish adding a missing step to fix a bug from removing an expensive step that never affected the outcome [2604.23853].

## 1. ClawTrace and TraceCards as the substrate

CostCraft depends on ClawTrace, an OpenClaw-native plugin that instruments an agent session through eight event hooks: `session_start`, `session_end`, `llm_input`, `llm_output`, `before_tool_call`, `after_tool_call`, `subagent_spawning`, and `subagent_ended` [2604.23853]. The tracing layer records every LLM call, every tool call, every sub-agent spawn and its nesting within the call tree, and per-span token counts and USD cost, including cache-aware billing. Two design choices are foundational. First, ClawTrace reconstructs a multi-agent call graph by mapping `childSessionKey → parentSpanId`, so the trace is a call tree rather than a flat event list. Second, its cost accounting distinguishes `input`, `output`, `cacheRead`, and `cacheWrite` tokens, because counting cache reads at full cost would overstate cost by `1.6–2×` and mis-rank steps.

The per-step cost formula is given as
\[
\mathrm{cost} = r_{\mathrm{in}} \cdot t_{\mathrm{in}} +
r_{\mathrm{out}} \cdot t_{\mathrm{out}} +
r_{\mathrm{cacheRead}} \cdot t_{\mathrm{cacheRead}} +
r_{\mathrm{cacheWrite}} \cdot t_{\mathrm{cacheWrite}},
\]
where the \(r_*\) terms are per-token USD rates and the \(t_*\) terms are token counts [2604.23853]. This cache-aware accounting is not an implementation detail but a prerequisite for later pruning logic, because prune decisions target high-cost spans.

A TraceCard is the compact intermediate representation CostCraft consumes. It is described as a deterministic YAML summary of one agent session, typically `1.2–1.8 kB`, small enough to fit dozens into an LLM context. Core fields include `total_cost_usd`, `total_tokens`, `top_cost_spans`, `redundant_tool_calls`, `sub_agents`, and `failed_or_repaired`. `top_cost_spans` stores the top-5 most expensive spans together with `kind`, `role_hint`, token counts, `cost_usd`, and `args_sample`. `redundant_tool_calls` clusters repeated tool calls with the same name and `≥0.8` Levenshtein similarity in their arguments. This representation is deliberately compact: CostCraft does not require raw spans or dashboards, only structured summaries [2604.23853].

## 2. Distillation pipeline and patch taxonomy

CostCraft has three stages: capture, compile, and distill. In the capture stage, the agent is run on tasks without skills and traced with ClawTrace. In the compile stage, each session is turned into a TraceCard. In the distill stage, TraceCards together with the current `SKILL.md` are fed to LLM-based analysts that emit patches, which are then merged into a new skill document [2604.23853].

The pipeline separates trajectories by outcome. Successful or partially successful runs are sent to a Success Analyst, while failed or partially failed runs are sent to an Error Analyst. From each TraceCard, the system may derive up to two patches: a preserve patch and an optional prune patch from successful trajectories, or a repair patch from failed trajectories. The merge operator resolves conflicts with the priority order `repair > prune > preserve` [2604.23853].

The three patch types are intentionally disentangled.

**Preserve patches** encode behaviors that helped produce a successful outcome. They are generated only from successful trajectories. To emit a preserve patch, the analyst must identify a behavior that helped achieve success, phrase it as a general principle rather than a task-specific instruction, and reference which step in the TraceCard demonstrates that behavior. Preserve patches are the lowest-priority patch type, singleton preserve patches are dropped, and preserve rules that appear in multiple trajectories are kept [2604.23853].

**Prune patches** remove or avoid steps that are expensive and did not affect the outcome. They are also mined from successful trajectories, not failures. The design is explicitly about efficiency rather than correctness. A prune patch is admitted only if it satisfies three constraints: it must target a named high-cost step from `top_cost_spans`, provide a natural-language counterfactual explaining why removing or avoiding that step would not change the result, and describe a behavior to avoid rather than a numeric budget such as “limit to \$0.05” [2604.23853].

**Repair patches** fix failure modes using oracle evidence. They come from failed or partially failed trajectories and are produced by the Error Analyst through a ReAct-style loop with the tools `inspect_mismatches`, `read_gold_snippet`, and `final_patch`, under a maximum of three tool calls. If diagnosis fails within that bound, the analyst emits a low-confidence patch that the merger downweights [2604.23853].

The paper’s error taxonomy on SpreadsheetBench assigns failure categories to repair or preserve rather than prune. `T1: No deliverable`, `T2: Wrong content type`, `T3: Formula not evaluated`, `T4: Placeholder mismatch`, `T5: Case/whitespace mismatch`, and `T7: Precision / rounding` map to repair, while `T6: Logic error` is handled through preserve. The fact that none of these categories map to prune is central: pruning is defined as removal of waste in already successful runs, not as a mechanism for fixing failures [2604.23853].

## 3. Cost attribution and counterfactual pruning

CostCraft’s central methodological claim is that per-step cost changes what the system can infer from trajectories. Without such attribution, a distillation pipeline sees only structural traces and final outcomes; with it, the system can reason about whether a successful run contains expensive but non-causal steps. This claim is operationalized through `top_cost_spans`, `redundant_tool_calls`, and the counterfactual gate applied to prune patches [2604.23853].

The paper formalizes prune reasoning at the step level. For a trajectory with steps \(s_1,\dots,s_n\) and step costs \(c_i\), total cost is
\[
C = \sum_i c_i.
\]
A prune patch targeting step \(s_k\) asserts that \(s_k\) is high-cost and that removing or changing it would leave the quality score unchanged:
\[
Q_{\text{with } s_k} = Q_{\text{without } s_k},
\]
while reducing cost:
\[
C_{\text{without } s_k} = C - c_k < C.
\]
This does not constitute a formal proof of non-causality. Rather, the system operationalizes “did not matter” through a constrained reasoning process: start from a successful run, inspect `top_cost_spans` and `redundant_tool_calls`, examine whether equivalent information was already available or whether the span’s output was used downstream, and require the analyst to articulate a counterfactual explanation [2604.23853].

The counterfactual gate, or CF gate, is therefore a structural constraint on what counts as an admissible prune patch. If a candidate prune patch does not name a top-cost span and provide a meaningful counterfactual justification, it is excluded from the merged skill in Full CostCraft. The reported ablation suggests that the quality and specificity of these counterfactuals matter: disabling the CF gate increases regressions in a small-sample test [2604.23853].

Two illustrative prune patterns are emphasized. One is repeated reading of the same input file, where TraceCards show high-cost duplicated `read_file` calls and `redundant_tool_calls` clusters. The corresponding prune rule is to read each input file once and cache its content rather than re-reading it. The other is unnecessary reading of workspace-memory files such as `MEMORY.md` or `SOUL.md` when the task is self-contained, producing the prune rule to skip such reads if the instructions already contain all necessary information [2604.23853].

A common misconception is that prune patches are merely hard budget constraints. The system explicitly rejects this interpretation. Prune patches must specify what behavior to avoid, not a numeric ceiling. Another misconception is that pruning is synonymous with aggressive cost minimization. The experimental findings suggest a narrower role at the studied scale: prune rules often act as guardrails that protect quality, even when they do not yet deliver large aggregate cost compression [2604.23853].

## 4. Experimental design on SpreadsheetBench

The primary benchmark is SpreadsheetBench, a set of `912` real-world spreadsheet-manipulation tasks where the agent must output a spreadsheet file and is evaluated by deterministic cell-level grading [2604.23853]. The quality metric is
\[
Q \in [0,1],
\]
defined as the fraction of rubric items satisfied, with \(Q = 1.0\) indicating exact cell-by-cell match. From the `200`-task professional subset, the study samples `50` tasks stratified by difficulty, runs each without skills, and records trajectories with ClawTrace. The split is `10` tasks for pipeline development, `10` tasks as the evolve set used to train the CostCraft skill, and `30` tasks as the held-out evaluation set. All runs use `openai-codex/gpt-5.4` via OpenClaw, `seed: 0`, and a sanitized workspace per run to avoid cross-episode leakage [2604.23853].

The ablation study compares five conditions: baseline with no skill, Full CostCraft, No-prune, No-cost-attribution, and No-CF on a 15-task subset. The held-out evaluation defines a regression as
\[
Q_{\text{skill}} < Q_{\text{baseline}} - 0.01,
\]
and a win as
\[
Q_{\text{skill}} > Q_{\text{baseline}} + 0.01.
\]
Cost change is measured through `total_cost_usd` from ClawTrace [2604.23853].

On the `30` held-out SpreadsheetBench tasks, the principal findings are negative in the narrow sense of cost compression and positive in the sense of regression control. Removing cost fields from TraceCards in the No-cost-attribution condition yields median cost uplift on successful tasks of `+49%`, compared with `+22%` under Full CostCraft, and increases regressions from `4` to `6`; `5` of those `6` regressions are catastrophic no-deliverable failures with \(Q=0\), compared with `1` under Full CostCraft [2604.23853]. This indicates that cost attribution is not merely an optimization aid; it changes the failure profile of the distilled skill.

The No-prune ablation is even more revealing. When prune rules are discarded, regressions rise from `4` to `13`, while median cost remains similar (`+15%` versus `+21%`). `8` of the `13` regressions produce no output at all [2604.23853]. The paper therefore interprets prune rules as protective even before they become strongly compressive. This is a notable departure from an intuitive expectation that prune rules would primarily affect cost.

The aggregate quality-preservation figure is also reported explicitly: Full CostCraft preserves quality on `86.7%` of held-out tasks, whereas stepwise removal of signals reduces preservation to `56.7%` under No-prune [2604.23853]. The study also notes that aggregate medians obscure important per-task structure: all skill conditions show net positive median cost change, yet this coexists with two dramatic recoveries, \(Q: 0 \to 1.0\) and \(0 \to 0.84\), alongside severe regressions in weaker ablations. This suggests that per-task and regime-partitioned analyses are more informative than a single cost average.

## 5. Transfer behavior and cross-benchmark asymmetry

A second benchmark, SkillsBench, is used to probe transfer. SkillsBench contains diverse tasks in data processing, scientific computing, document analysis, code, and related areas, graded by Docker-contained `pytest` tests with binary pass/fail outcomes. None of the selected tasks are spreadsheets [2604.23853]. The transfer experiment evaluates `30` SkillsBench tasks under the baseline with no skill and under the SpreadsheetBench-trained Full CostCraft skill.

The most important cross-benchmark finding is an asymmetry between prune and preserve rules. On SkillsBench, cost decreases on `16` of `27` valid task pairs, and median per-task cost falls from `\$0.105` to `\$0.071`, a `32%` median cost reduction [2604.23853]. At the same time, there are `3` regressions—`edit-pdf`, `energy-market-pricing`, and `invoice-fraud-detection`—in which the baseline passes and the CostCraft skill fails. The reported root cause is that preserve rules encode spreadsheet-style conventions that conflict with SkillsBench verifiers. There are also `2` quality wins—`citation-check` and `glm-lake-mendota`—where the baseline fails and the CostCraft skill passes.

The paper interprets this as evidence that prune rules targeting universal waste patterns transfer well, whereas preserve rules trained on benchmark-specific conventions can cause regressions on new task types [2604.23853]. This is one of the central conceptual outcomes of the work. CostCraft does not merely divide knowledge into “successful” versus “failed” behaviors; it separates “what to keep” from “what to cut,” and these two categories exhibit different transfer properties.

At the scale studied, only two prune rules were learned: skip reading workspace-memory files when the task is self-contained, and read each input file once and cache rather than re-reading it [2604.23853]. On the `17` successful held-out SpreadsheetBench tasks, only `2` tasks, or about `11.8%`, match these waste patterns. On those two tasks, Full CostCraft still increases cost (`+38%` and `+22%`), but No-prune causes one to fail entirely with \(Q=0\). This again supports the interpretation that, at this training scale, prune rules serve primarily as quality-preserving structure rather than direct cost compression.

## 6. Implementation, limitations, and significance

ClawTrace is implemented as an OpenClaw plugin whose events are batched in memory and sent once per session to `/v1/traces/events` in JSON form. A backend pipeline built on PuppyGraph and Iceberg materializes spans into tables. The reported tracing overhead is about `0.30%` of agent wall-clock time, with median `445 ms` flush time compared with `147 s` trajectories, and there is no quality divergence in on/off comparisons on `10` tasks [2604.23853]. This low overhead matters because CostCraft presupposes routine trace collection.

The merge product of CostCraft is an evolved `SKILL.md` organized around sections such as Trigger, Workflow, Stop rules, Artifact checklist, and Cost control, with the latter populated by prune rules [2604.23853]. This document-centered representation is significant because it positions distillation as structured editing rather than parameter updates or policy optimization. A plausible implication is that CostCraft is designed for environments where prompt- or document-level skills are the control surface.

The work has several explicit limitations. It is evaluated with a single backbone, `openai-codex/gpt-5.4`, and a single seed; there is no multi-seed variance analysis. The evolve set is small—`10` tasks—yielding only `2` learned prune rules. The held-out evaluations are also limited to `30` SpreadsheetBench tasks and `30` SkillsBench tasks. Some heuristic TraceCard fields, such as `sub_agents.output_used_in_final`, are under-tested because no sub-agents were spawned in the experiments. The failure taxonomy was coded by a single annotator, and SkillsBench includes only the full-skill transfer evaluation rather than full ablations such as No-prune or No-cost-attribution [2604.23853].

The broader significance claimed for CostCraft is not that it already yields universal cost savings in every setting. Rather, it introduces a cost-aware intermediate representation, a three-way patch typology grounded in cost, and a counterfactual pruning requirement that ties efficiency interventions to named high-cost steps [2604.23853]. The paper further releases ClawTrace and TraceCards as open infrastructure for cost-aware agent research. This suggests a more general research agenda in which trajectory distillation, observability, and cost optimization are treated as a single pipeline rather than separate concerns.

In that sense, CostCraft defines a specific methodological stance within LLM-agent research: successful trajectories contain not only evidence of what should be preserved, but also evidence of what can be safely removed, provided that per-step cost and counterfactual structure are available. The experimental asymmetry between prune and preserve transfer sharpens that stance. Efficiency knowledge appears to be more portable when it targets universal waste patterns such as redundant reads, while correctness knowledge tied to local task conventions may be brittle outside its source benchmark [2604.23853].

Source: https://www.emergentmind.com/topics/costcraft