Compression, structure, and executor capability: a controlled real-cost decomposition of language-model agent skill optimisation

Abstract: Agent skills, reusable instruction artefacts supplied to a tool-using LLM, are increasingly optimised by shortening, structural rewriting, stronger-model compilation, and scoped loading, on the assumption that a smaller or better-organised skill lowers cost while preserving success. That assumption is rarely tested with quality and real monetary cost measured on the same runs and the contributing factors separated. This study reports a controlled decomposition over ten skill-delivery conditions, 40 software-engineering tasks, and three repetitions per cell (1,200 rollouts), separating no-skill execution, raw skills, deterministic shortening, linear and structured rendering from a shared semantic ledger, scoped loading, and the compiler and executor model tiers. Quality is the verifier pass rate at task level; cost is solve-stage token cost at standard provider prices, with a token-volume-normalised view for robustness and compilation cost amortised separately. The task is the unit of inference, intervals are task-clustered, and the contrast family is multiplicity-controlled. Deterministic shortening is close to the raw baseline but does not establish non-inferiority within the preset margin. Structured rendering and scoped loading lower pass rate on the compact executor without lowering cost, and structured rendering is indistinguishable from linear text at matched content. The only contrast surviving correction is executor capability, which raises pass rate by 27 percentage points at roughly five times the real cost, with compiler tier showing no robust effect. Under real prices no optimised representation reaches a practical break-even. The evidence indicates that executor capability is the dominant lever and that no representation strategy improves over the raw skill on either executor tier.

• The paper demonstrates that using a strong executor increases pass rates by 26.7 percentage points compared to a compact executor, outclassing traditional skill optimizations.
• It finds that deterministic shortening gains only a marginal 1.7% improvement with negligible cost savings, while structured rendering and scoped loading lead to lower task success.
• Cost analysis shows that high-tier executors incur five- to sevenfold higher real costs despite reduced token counts, highlighting the need for currency-based cost evaluation.

Controlled Decomposition of LLM Agent Skill Optimisation: Cost and Quality Analysis

Introduction

The paper "Compression, structure, and executor capability: a controlled real-cost decomposition of language-model agent skill optimisation" (2607.03048) presents a systematic experimental study on the efficacy of agent skill optimisation techniques for language-model-driven automation, focusing on the trade-offs among skill compression, structure, and executor model capability. The investigation isolates the effects of several widely employed representation strategies—skill shortening, structural rewriting, model-tier compilation, and scoped loading—on both quality and actual monetary cost under matched experimental settings for LLM-based software agents.

This analysis is motivated by the increasing prevalence of "externalization," whereby reusable agent skills are represented as explicit artefacts and passed to LLMs, despite persistent resource constraints in context windows, inference latency, and financial cost. Contrary to common assumptions, the study rigorously tests whether more compact or more structured skill representations provide the expected cost savings without degrading task performance, using a multi-condition, multi-task, and multi-repetition experimental platform featuring two model tiers and programmatically verifiable outcomes.

Experimental Design

The study executes 1,200 programmatically measured rollouts covering 10 skill-delivery protocols, 40 diverse software-engineering tasks, and three runs per cell. The main experimental axes include:

• Skill Representation: Raw curated skill, deterministic extractive shortening, linear and structured renderings generated from a unified semantic ledger, and scoped loading.
• Compiler and Executor Model Tier: Compact (gpt-5.4-mini) and strong (gpt-5.5) models used for both offline compilation and online execution in a factorial arrangement.
• Parallels and Controls: Content-holding controls for equalising the payload between linear and structured renderings, and strict exclusion of task-specific answers from the compilation process.

Quality is measured as pass rate according to executable verifiers; cost is grounded in actual provider rates for tokenised solves, normalized to control for caching and normalization artifacts.

Results: Effects on Task Success and Cost

The raw-skill baseline lifts pass rate from 26.7% (no skill) to 42.5% under a compact executor. The strong executor acting on the raw skill yields a 64.2% pass rate, clearly outperforming all protocol-level optimizations in both quality and reward.

Figure 1: Pass rate by condition with task-clustered 95% intervals.

When contrasting skill optimisation strategies with the raw baseline:

• Deterministic Shortening: Achieves a pass rate difference of +1.7 percentage points, with confidence intervals encompassing the non-inferiority margin; real cost difference is negligible.
• Structured Rendering and Scoped Loading: Result in negative pass rate differentials (−13.3 and −8.3 percentage points, respectively) with no practical cost reduction. Notably, structured rendering performs equivalently to linear presentation when content is held constant.

Figure 2: Paired pass-rate contrasts with task-clustered 95% intervals.

• Compiler/Executor Tier: Upgrading the executor is the dominant effect; switching from compact to strong within structured rendering increases pass rate by 26.7 percentage points (significant at Holm-adjusted $p=0.003$), but at a real cost increase of approximately fivefold per task. Changing the compiler (offline skill-generator) does not materially impact quality.
• Resource Utilization: Although skill representation strategies do reduce the injected context footprint (e.g., raw skill ≈4,550 tokens, structured ≈2,500–3,000, scoped loading ≈2,080), this does not translate to a reduced token count at solve or to real cost savings. Across compact-executor conditions, actual tokens generated range from 46,000 to 51,000, and costs are tightly clustered.

Figure 3: Real and token-volume-normalised cost per task by condition.

• Real Price vs. Normalized Costs: Results show that normalizing token volume creates misleading impressions of resource savings by high-tier models. When priced at actual provider rates, strong executors cost five to seven times more than compact ones per task, notwithstanding sometimes lower token usage.
• Compilation Amortization: Offline model compilation costs for skill transformation are not recouped in practice; no optimised representation reaches a practical break-even for most tasks, with the median task requiring over 150 reuses to offset the cost relative to the raw baseline.

Implications and Interpretation

This controlled decomposition provides contrary evidence to the commonly held belief that skill compression, scoping, or structural rewriting yield efficient, quality-neutral agent operation. The primary and only statistically robust determinant of improved pass rate and outcome quality is executor capability. Representation optimizations—shortening, structuring, and scoped loading—neither significantly enhance quality nor generate meaningful financial efficiency on the deployed agent platform.

These results have immediate methodological implications:

• Cost Evaluation: Cost savings claims in LLM agent literature must be framed in actual currency at real provider rates, rather than tokens, to accurately reflect the utility-scale impact.
• Executor Selection: Investments in model selection (higher-capability executors) have orders-of-magnitude stronger effect on agent performance than prompt manipulation or skill-refactoring.
• Representation Strategy: Externalized skill refinement, absent evidence of content loss or bespoke environment constraints, should not be expected to independently deliver significant improvements.

Practically, these findings recommend that efforts and resources for agent-fielded optimization concentrate on leveraging the best available execution model rather than elaborate artefact engineering. The lack of practical break-even for any static pre-optimisation protocol challenges the theoretical motivation for current skill compilation pipelines.

Future Developments

Potential extension paths include evaluating these findings on alternate LLM provider infrastructures, more diverse or more granular task corpora, and in real-world deployment cycles with persistent agent memory and dynamic skill evolution. There is also room to investigate whether hybrid approaches coupling static skill representation and runtime learning could realize efficiencies beyond those measured here.

Fundamentally, the evidence underscores the need for cost-accounted, decomposition-driven methodology in agent benchmarking, and for aligning optimization claims with operational financial realities.

Conclusion

This study decisively demonstrates that, within the prevailing experimental regime, executor capability is the chief determinant of software agent quality, and that common forms of skill artefact optimization—deterministic shortening, structured rewriting, scoped loading, and offline model-tier compilation—provide no measurable benefit in pass rate or cost efficiency over simply delivering raw curated skills. These results call for caution against attributing agent improvement to representation-level interventions and emphasize upfront, currency-based cost modeling as integral to agent evaluation (2607.03048). Expanding the scope to other task domains and model families will further illuminate the generality of these conclusions.