Total Cost of Agency: Exact Attribution of Memory Injection Cost in Multi-Agent LLM Workflows
Abstract: Every node in a multi-agent LLM workflow retrieves context from memory and injects it into its prompt, where those injected tokens are billed as input tokens at the same per-token price as the system prompt and the user query. Production observability tools report total token cost but do not separate the tokens a node generates from the tokens it is handed, so this component of the bill is invisible to the teams paying it. We introduce the Total Cost of Agency (TCA), a decomposition of multi-agent workflow cost into base prompt, inference, memory injection, miss penalty and context-accumulation components, and an exact attribution method: a two-pass, non-billable token count that measures injected tokens directly rather than estimating them from word-count proxies. On a 200-task enterprise benchmark executed against real model APIs, memory injection accounts for 13.6 percent of the variable cost a compile-time optimizer can act on, about 12 percent of the full billed cost, and its share rises from a structural zero at workflow depth one to 27.6 percent at depth six. Injected tokens grow linearly with depth over the measured range (R2 = 0.9974, depths two through six); a quadratic fit yields a negative leading coefficient, so the data do not exhibit convex growth at these depths. We show the component is controllable at fixed model tier: reducing the retrieval window capacity from 32 to 2 entries lowers injected tokens by 28.7 percent with an accuracy change within seed-level variation. We report in full that our graph-rewriting transforms are approximately cost-neutral in isolation, that two of the five decomposition terms are zero by construction in this harness, and that total workflow cost is dominated by model tier assignment, which we hold fixed and treat as prior work. Prompt caching is not evaluated; all figures are for the uncached case.
Summary
- The paper introduces the Total Cost of Agency (TCA), a framework for decomposing workflow cost of multi-agent LLM systems into five terms to separately attribute memory injection costs.
- The proposed exact token-level attribution method reveals that memory injection contributes 13.6% of variable cost, 11.8% of full billed cost in variable model pricing, and up to 27.6% in deeper workflows.
- The paper suggests reducing memory capacity as a cost control mechanism, demonstrating that decreasing warm memory capacity by 98% reduced costs by 4.1%.
Problem formulation and contribution
“Total Cost of Agency: Exact Attribution of Memory Injection Cost in Multi-Agent LLM Workflows” (2609.23790) isolates a cost component that conventional LLM observability systems generally aggregate into a node’s total input-token count: the tokens retrieved from memory and injected into the node’s prompt. This distinction matters in multi-agent workflows because each node may consume outputs produced by upstream nodes, while providers bill those outputs again when they are supplied as input context. Existing cost analyses typically emphasize model-tier selection, output generation, or total prompt length; they do not separately attribute the portion of input tokens created by memory operations.
The paper’s central proposal is the Total Cost of Agency (TCA), a decomposition of workflow cost into five terms: base-prompt cost, inference cost, memory-injection cost, miss penalty, and context-accumulation cost. The decomposition is intended to distinguish directly billed components from deployment-dependent terms. In the evaluated configuration, base-prompt, inference, and memory-injection costs are billed, whereas miss penalties and accumulation are zero as independent monetary terms by construction. In particular, the paper treats inherited context and retrieved memory as observationally indistinguishable once incorporated into a prompt; consequently, the accumulation term is conceptual rather than separately measurable under the proposed instrumentation.
The empirical claim is deliberately narrower than a general optimization claim. The paper holds model-tier assignment largely fixed, acknowledges that routing dominates total cost, and investigates whether memory injection is measurable, structurally depth-dependent, and controllable through retrieval capacity. Its strongest contribution is therefore accounting and attribution rather than a new routing policy or memory architecture.
Exact token-level attribution
The proposed measurement method assembles each node prompt twice. The first prompt contains only the base components: the specialist system instruction, tool definitions, and user query. The second adds retrieved memory. Both prompts are tokenized with the provider’s tokenizer, and the difference in token counts is attributed to memory injection:
This is an attribution procedure rather than a predictive cost model. It measures the prompt that will actually be dispatched, avoiding the systematic errors of word-count and character-count proxies. Those errors are especially relevant for enterprise workloads, whose outputs contain identifiers, SQL, numerical literals, structured records, and other material with tokenizer behavior that differs substantially from ordinary prose.
The method has favorable operational properties. The counts are non-billable, add approximately 10 milliseconds per node, and operate against inference latencies reported between 80 and 3,000 milliseconds. The base-prompt token count is obtained as a by-product. The instrumentation therefore exposes both the absolute injected volume and its monetary value at the tier actually selected for the node.
The causal interpretation remains limited. The procedure establishes how many tokens memory contributed to a prompt, but not whether those tokens improved task performance or whether removing them would change the output. Those questions require paired executions under different memory policies.
TCA and workflow-depth behavior
The paper models a workflow as a directed acyclic graph in which nodes execute in topological order and retrieve prior outputs under a capacity-bounded or unbounded memory strategy. For a linear chain with unbounded history, cumulative injection can grow quadratically with depth because each node receives all preceding outputs. Under a bounded retrieval window, however, the per-node injected history is capped, producing approximately linear total growth once the workflow exceeds the capacity regime.
The benchmark contains 200 enterprise tasks distributed evenly across five categories and five workflow depths from two through six. The categories are billing reconciliation, software asset management, identity and access management, cross-domain reconciliation, and policy checking. Each category-depth cell contains eight tasks. Workflows use specialist nodes for extraction, query generation, policy checking, billing reconciliation, IAM auditing, and cross-domain reconciliation, backed by seeded SQLite databases.
At the mid tier, the unoptimized baseline records an average of $0.020408 per task in inference cost and$0.003220 in memory-injection cost. Thus, injection represents 13.6% of the variable cost considered actionable by a compile-time optimizer. The paper also derives an estimated base-prompt cost of $0.003699 per task, based on 304.4 base tokens per node and a four-node mean workflow. Under this derived denominator, injection constitutes approximately 11.8% of full billed cost. The latter figure is explicitly identified as an estimate rather than a direct per-condition measurement.
The absolute baseline token counts rise from 143 injected tokens per task at depth two to 754 at depth six. A linear fit over depths two through six has $R^2 = 0.9974R^2-4.57$. Accordingly, the observed data do not support convex growth over the measured range; they are concave over these five depths. This result is important because the paper’s theoretical discussion motivates quadratic growth only for unbounded history, whereas the experiment uses bounded retrieval.
Figure 1: Injection volume and injection-cost share across workflow depth; depth one is a structural zero and is excluded from the regression.
The depth-one value is not an empirical zero in the same sense as the other observations. A source node has no upstream output to retrieve, so zero injection follows from graph structure. Nevertheless, instrumentation records exactly zero injected tokens across 1,112 extraction-node observations. The injection share of total cost increases from 8.4% at depth two to 27.6% at depth six on the mid tier, and reaches 28.0% on the small tier. The near-overlap of the tier-specific curves follows from the provider price schedule, where input and output prices have the same ratio across the two tiers. The conclusion is therefore conditional on that pricing structure: token volumes transfer directly, but cost shares need to be recomputed under other provider price ratios.
The result establishes that injection is structurally associated with workflow depth even though inference remains the dominant cost component. It also shows why reporting only aggregate node input tokens obscures a workflow-level cost driver: the memory contribution increases with the dependency structure rather than merely with the local task complexity.
Retrieval capacity as a controllable lever
The most informative intervention varies retrieval capacity while holding tier assignment fixed. Reducing the warm-memory capacity from 32 entries to 2 decreases mean injected tokens from 1,074 to 766 per task, a 28.7% reduction. Cost per task falls from $0.007436 to$0.006939, or 6.7%.
The token-level comparison is the stronger result because both conditions use the same tier assignment. It demonstrates that the reduction in injected volume is attributable to the memory mechanism rather than to routing. The directly attributable token reduction corresponds to $0.000308 per task, or approximately 4.1% of total cost. The larger measured cost reduction is not assigned entirely to memory injection: shrinking the context may also reduce output length, and output tokens were not instrumented against retrieval capacity.
Accuracy changes from 0.600 to 0.570 in this single-seed comparison. The paper does not interpret this difference as a demonstrated degradation because it lies within the range observed across single-seed ablation conditions. This is a reasonable qualification, but it also means that the experiment does not identify the capacity-performance frontier. It shows that capacity is an effective cost lever, not where the acceptable operating point lies.
Negative results and separation from routing
The graph-rewriting interventions do not produce a meaningful cost reduction in isolation. At the small tier, conditions A through G range from $0.00740 to0.00998 per task compared with$0.00742 for the unoptimized baseline. This apparently adverse result is explained by the tier assigner: the small tier is already the price floor, so accuracy-sensitive nodes can only be escalated. The result prevents an incorrect interpretation of tier-dependent dollar reductions as memory savings. In an earlier comparison, condition H lowers the injection-cost line in dollars, but this reflects reassignment to the cheaper tier for most nodes rather than a reduction in injected-token volume.
The paper therefore separates two optimization axes. Memory capacity controls how many tokens are injected. Tier assignment controls the price applied to both injected and generated tokens and has a much larger effect on total cost. The study makes no claim that memory optimization supersedes routing methods such as FrugalGPT (Chen et al., 2023), RouteLLM, or MasRouter. Instead, it supplies an accounting layer that could expose memory costs to those routing and compilation systems.
Limitations and open questions
The principal limitation is the absence of prompt-caching measurements. Cached input can be substantially cheaper than uncached input, but the paper’s argument that dynamically generated memory may be a poor caching candidate is structural rather than empirical. Upstream outputs are often written for the first time within a task, node-specific system prompts can invalidate reusable prefixes, cacheable-prefix thresholds may exceed the injected context, and writes with few subsequent reads can cost more than they save. None of these claims substitutes for cache-hit-rate measurements. The paper explicitly leaves open the residual injection cost under realistic caching policies.
The benchmark also uses fixed-topology DAGs rather than ReAct-style loops or dynamically branching workflows. The theoretical mechanism should apply to iterative agents, but the magnitude of injection cost in such systems is not measured. Since loops can accumulate observations over many iterations, the depth-six results cannot be treated as estimates for long-horizon agents.
External validity is constrained by the use of one provider, one framework, two active model tiers, and synthetic-but-realistic SQLite enterprise data. The token-counting method should transfer to systems that expose explicit prompt assembly and a provider tokenizer, but the measured magnitudes depend on pricing, tokenizer behavior, prompt templates, output lengths, and workflow topology.
The accuracy evaluation is also limited. Query-generation correctness combines execution success, nonempty results, and required-topic recall, while other nodes use topic recall alone. The benchmark contains 40 tasks per category, ablation and capacity results use one seed, and the reported accuracy intervals are too broad to support fine-grained claims. The specific missing experiment is a multi-seed sweep over capacities , reporting token volume, cost, and accuracy jointly. Without that sweep, the paper establishes the existence and direction of the capacity trade-off but not its knee.
Conclusion
The paper establishes memory injection as a separately measurable component of multi-agent LLM workflow cost. Its two-pass tokenizer-based method provides exact attribution with negligible operational overhead, and the benchmark shows that injection accounts for 13.6% of optimizer-controllable variable cost, approximately 12% of full billed cost, and up to 27.6% of cost at depth six. Injection grows nearly linearly over the measured bounded-history regime, while graph rewrites are effectively cost-neutral and model-tier assignment remains the dominant cost lever.
The principal unresolved issue is how much of this cost survives prompt caching and how retrieval capacity should be selected under a properly powered accuracy-cost evaluation. Until those measurements are available, the paper’s strongest result is an accounting claim: memory-generated context is billed, depth-dependent, and invisible unless it is instrumented separately.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is this paper about?
This paper studies how much money is spent when AI agents pass information to one another in a multi-step workflow.
Imagine a team of AI workers:
- One AI reads some information.
- Another AI uses that result to write a database query.
- A third AI checks the answer.
- A final AI makes a decision.
Each AI receives information from earlier steps. This shared information is called memory injection or injected context. The paper argues that this information costs money because it becomes part of the next AI’s prompt, and prompt tokens are billed by AI companies.
The main idea is that current monitoring tools show the total cost of an AI call, but they usually do not show which part of the cost came from information supplied by earlier agents.
The authors introduce a measurement system called Total Cost of Agency, or TCA, to track this cost more carefully.
2. What questions are the researchers asking?
The paper focuses on several simple questions:
- How much of an AI workflow’s bill comes from memory passed between agents?
- Does this cost increase as the workflow has more steps?
- Can the memory cost be measured exactly rather than guessed?
- Can companies reduce this cost by limiting how much past information each agent receives?
- Does reducing memory hurt the system’s accuracy?
- Are other possible improvements, such as changing the workflow’s structure, actually useful for saving money?
The paper is not mainly trying to discover which AI model is cheapest. The authors explain that choosing a cheaper or more suitable model usually has a much larger effect on total cost. Instead, they study memory-related costs as a separate part of the bill.
3. How did the researchers study the problem?
The test workflows
The researchers tested 200 enterprise-style tasks. These tasks involved areas such as:
- Billing and invoice checking
- Software asset management
- Identity and access management
- Policy checking
- Cross-domain data reconciliation
The workflows had between two and six steps. Each step was handled by a specialist AI agent, such as an extraction agent, query-writing agent, or policy-checking agent.
The data came from three SQLite databases containing realistic but artificial business information, such as invoices, purchase orders, payments, and access records.
Understanding tokens
AI systems do not read text exactly as humans do. They break text into small pieces called tokens.
For example, a long word, number, or computer command may be split into several tokens. AI companies charge for these tokens, so counting them accurately matters.
A simple analogy is counting the cost of sending messages by characters or postage stamps. If you only estimate the number of words, you may get the price wrong because different words take different numbers of stamps.
The two-pass counting method
To measure memory injection, the authors prepare each agent’s prompt twice:
- Once with only the normal information, such as instructions and the user’s question.
- Once with that same information plus the memory retrieved from earlier agents.
They then count the tokens in both versions. The difference is the number of tokens added by memory.
In simplified form:
1 2 |
injected memory tokens = tokens in full prompt − tokens in basic prompt |
The counting uses the AI provider’s own tokenizer, so the researchers measure the tokens that are actually likely to be billed.
This process does not ask the AI to generate an answer. It only counts tokens, so it is not billed as an ordinary AI request. It adds about 10 milliseconds per agent, which is very small compared with the usual AI response time.
The cost categories
The paper divides workflow costs into five parts:
| Cost part | Meaning |
|---|---|
| Base prompt | Instructions, tools, and the user’s question |
| Inference | The tokens generated by the AI |
| Memory injection | Information added from earlier agents |
| Miss penalty | Extra cost if the system must search a slower memory store |
| Accumulation | The cost of carrying older information through the workflow |
In the experiments, the last two categories were zero in terms of money. The memory store was local, so a failed quick lookup caused extra time but not an extra token charge.
4. What did the researchers find?
Memory injection is a meaningful part of the bill
For the main mid-level model condition:
- AI-generated output was the largest cost.
- The base prompt and injected memory were smaller but still important.
- Memory injection made up about 13.6% of the cost that the tested optimizer could change.
- It was estimated to be about 11.8% of the full bill.
In one example, each task used approximately:
- 1,233 base-prompt tokens
- 1,073 memory-injection tokens
This means that memory was almost as large as the original prompt information.
The important point is not that memory was the largest expense. It was not. The important point is that this cost was previously hidden inside the general input-token count.
Memory costs rise with workflow depth
The deeper the workflow became, the more information later agents received.
The average number of injected tokens per task increased approximately as follows:
| Workflow depth | Injected tokens |
|---|---|
| 2 | 143 |
| 3 | 324 |
| 4 | 466 |
| 5 | 602 |
| 6 | 754 |
At depth one, the cost was zero because the first agent had no earlier agent’s output to receive.
The share of total cost caused by memory injection also increased:
- About 8.4% at depth two
- About 27.6% at depth six
This shows that memory becomes more important in longer chains of agents.
The authors expected that memory might grow quadratically, meaning it could grow faster and faster as workflows became deeper. However, in the depths they tested, the data looked more like a straight line. The researchers emphasize that this does not prove memory will always grow linearly, especially in much longer or more complicated workflows.
Limiting memory reduced the number of tokens
The researchers compared two memory limits:
- A window that could keep up to 32 entries
- A smaller window that could keep only 2 entries
Reducing the window from 32 to 2 entries:
- Lowered injected tokens by 28.7%
- Lowered the measured cost per task by about 6.7%
- Changed accuracy from 0.600 to 0.570 in that test
The authors do not claim that accuracy truly became worse because the experiment used only one random seed, and the difference was within the amount of variation seen elsewhere.
Still, the result shows an important trade-off: giving agents less history can save money, but removing too much information may eventually hurt their answers.
Reorganizing the workflow did not save much money
The researchers tried several changes to the workflow, including:
- Combining some agent steps
- Reordering steps
- Sharing information in a different way
These changes moved information around, but they did not greatly reduce the amount of work. Their costs were within about 1% of one another.
The paper therefore does not find that these workflow rewrites are useful cost-saving methods by themselves.
Choosing the model tier mattered more
The authors found that the biggest effect on total cost came from which model each agent used.
A powerful model costs much more per token than a smaller model. Therefore, sending simple tasks to a small model and difficult tasks to a larger model can change the bill much more than memory management can.
The researchers deliberately kept model assignment mostly fixed so they could study memory separately.
Caching remains unanswered
AI companies sometimes offer cheaper prices when a prompt is reused through prompt caching. Caching is like keeping a frequently used book on a nearby shelf instead of fetching it from a distant library every time.
The paper discusses caching but does not test it. The researchers say that memory from earlier agents may be difficult to cache because:
- It is often created only once for a particular task.
- Different agents may use different instructions.
- Small pieces of memory may not be large enough to qualify for caching.
- Storing information that is never reused could cost more than it saves.
These are reasonable possibilities, but the authors clearly state that they have not measured cache hit rates.
5. Why is this research important?
The paper gives AI developers a more detailed way to understand their bills.
Instead of seeing only:
1 |
Total input tokens: 10,000 |
a developer could see something like:
1 2 3 |
Base instructions: 3,000 tokens Memory from other agents: 2,500 tokens User information: 4,500 tokens |
That makes it easier to decide what to change.
For example, a company could:
- Reduce how many previous results each agent receives.
- Compress old information before passing it along.
- Use shorter summaries instead of full outputs.
- Design prompts so that useful information can be cached.
- Choose cheaper models for simpler steps.
- Monitor whether saving tokens causes accuracy problems.
The paper’s measurement method is also useful because it counts tokens directly using the provider’s tokenizer rather than estimating them from word counts.
Conclusion
In simple terms, the paper shows that AI agents can become expensive not only because of the answers they generate, but also because of all the information they carry from earlier steps.
The researchers created a way to measure this hidden cost exactly. In their tests, memory injection made up a noticeable part of the bill and became more important as workflows became deeper. Limiting the amount of retrieved memory reduced the number of tokens used, although the best balance between savings and accuracy still needs more testing.
The broader lesson is that companies building multi-agent AI systems should track both:
- Which model each agent uses, and
- How much information each agent receives from the past.
Model choice is still the larger cost lever, but memory tracking can help make complex AI workflows more efficient and easier to manage.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
- Prompt caching remains unmeasured. The study does not report cache hit rates, cache-write costs, cache-read costs, prefix reuse, or residual billed injection cost under realistic caching policies.
- The effectiveness of cache-aware prompt design is unknown. It is unresolved whether restructuring prompts so that shared memory precedes node-specific instructions could substantially improve cache reuse without harming task performance.
- The cost–accuracy trade-off for retrieval capacity is not characterized. Only and are compared in the capacity experiment, at one seed; the location of the optimal “knee” is unknown.
- The effect of memory capacity on output length is not isolated. The reported cost reduction may partly result from shorter model outputs after context removal, but output tokens were not instrumented against retrieval capacity.
- Dynamic agent workflows are not evaluated. ReAct-style loops, retries with changing context, conditional branches, self-reflection, and workflows whose topology is generated during execution may exhibit substantially different injection-growth patterns.
- The claimed linear depth relationship is limited to a narrow regime. Measurements cover depths two through six under a bounded retrieval window, so they cannot establish behavior at larger depths, under unbounded history, or near context-window limits.
- General DAG structure is underexplored. The study does not separately quantify how fan-in, fan-out, parallel branches, merge nodes, heterogeneous path lengths, or duplicate context affect injection volume and cost.
- The relationship between graph topology and memory policy is unresolved. It is unclear which retrieval strategies—such as recency, relevance, dependency-aware selection, or deduplication—minimize injected tokens while preserving task quality.
- The memory-injection attribution method is not validated across provider-specific prompt semantics. The subtraction of token counts may behave differently when providers apply hidden formatting, message wrapping, tool-call serialization, system-level transformations, or provider-side tokenization rules.
- Reproducibility is constrained by missing implementation details. The paper does not provide the workflow code, prompt templates, retrieval implementation, exact model/API versions, tokenizer versions, generation parameters, retry behavior, or task data needed to independently reproduce the measurements.
- The benchmark’s external validity is uncertain. The 200 tasks use synthetic enterprise databases and five task categories, so the results may not transfer to production workloads with noisy records, larger databases, regulated data, long-lived memory, or user-generated inputs.
- The single-provider and single-framework design leaves cross-platform variation unknown. The magnitude of injection cost, tokenizer overhead, cache behavior, and prompt-format effects has not been tested across multiple providers, models, agent frameworks, or serving stacks.
- The observed tier-independence of injection-cost share may not generalize. The result depends on the particular input/output price ratios used; providers with different pricing structures could yield materially different cost shares and optimization priorities.
- The interaction between memory optimization and model routing is not studied. Tier assignment is held fixed, leaving unresolved whether reducing or compressing memory changes routing confidence, escalation frequency, model choice, or total cost.
- Memory compression and retrieval-quality methods are not compared. The study varies only retrieval-window capacity and does not test summarization, semantic compression, pruning, deduplication, or learned context selection as alternatives.
- The accuracy evaluation is too coarse to establish semantic preservation. Execution success and keyword recall at a fixed threshold may miss factual errors, incomplete reasoning, invalid SQL results, hallucinated policies, or subtle degradation caused by removing context.
- Accuracy uncertainty is insufficiently resolved for the key optimization result. The capacity experiment uses one seed and two settings, so the reported 0.030 accuracy difference cannot distinguish stochastic variation from a systematic effect.
- The statistical significance of small cost differences is unknown. Single-seed ablations lack variance estimates, making it impossible to determine whether the approximately one-percent differences among graph-rewriting conditions reflect real effects or measurement noise.
- The decomposition does not fully separate inherited context from retrieved memory. is defined conceptually but is zero in the measured configuration because inherited and retrieved tokens are not separately identifiable; deployments with both mechanisms require a more granular attribution scheme.
- Remote durable-memory costs are not empirically characterized. The miss-penalty component is zero because the durable store is local, leaving unresolved how network latency, remote storage pricing, serialization, and repeated retrieval affect TCA.
- The attribution method is not tested under prompt truncation or context-window overflow. It is unknown whether memory eviction, provider-side truncation, or failed requests create attribution errors or alter the relationship between retrieved and billed tokens.
- The operational overhead of continuous instrumentation is only estimated. The reported approximately 10 ms per node is not evaluated under high concurrency, batching, serverless execution, network contention, or production-scale observability pipelines.
- The study does not quantify long-term or multi-session memory effects. It focuses on within-task context accumulation and does not examine persistent memories, cross-session reuse, stale entries, memory maintenance costs, or deletion policies.
- The proposed accounting has not been connected to an end-to-end optimizer. The paper demonstrates measurement and one capacity lever but does not show whether TCA-based decisions produce reliable savings when jointly optimizing retrieval, compression, routing, caching, and workflow structure.
Practical Applications
Immediate Applications
The paper’s instrumentation and findings support the following applications that can be deployed with existing LLM APIs and agent frameworks.
- Per-node memory-cost observability for LLM production systems (software and enterprise AI)
- injected tokens and cost per node;
- memory cost by workflow depth and agent role;
- base, inference, injection, and miss costs;
- cost per task, category, tenant, or model tier.
- Dependencies: the framework must expose prompt assembly before inference, and the provider must provide a compatible tokenizer. Tokenizer behavior and provider billing rules must remain aligned.
- Cost-aware workflow profiling and budgeting (enterprise IT, finance, SaaS operations) Organizations can use the Total Cost of Agency decomposition to create budgets and alerts for multi-agent workflows. For example, a billing-reconciliation pipeline could trigger an alert when memory injection exceeds a specified percentage of task cost or when a workflow’s depth causes injection to exceed a per-task token budget. Dependencies: accurate provider prices, consistent logging, and a clear policy for whether base prompts, inference, and memory costs are charged to departments or customers.
- Retrieval-window tuning as an immediate cost-control experiment (RAG systems and agent memory)
- customer-support agents;
- software engineering copilots;
- IAM and compliance assistants;
- financial reconciliation agents;
- internal knowledge-management systems.
- Dependencies: the benchmark’s accuracy result is not conclusive; the reported capacity experiment used one seed and two settings. Each deployment must identify its own cost–quality “knee” and verify that important context is not evicted.
- Depth-based cost forecasting for workflow design (workflow engineering and agent orchestration) The measured increase from 143 injected tokens at depth two to 754 at depth six, together with the strong linear fit over the tested range, can be used to create an early-stage cost estimate for fixed-depth DAGs. Architects can compare alternative designs before deploying them, such as a four-node chain versus a shallower workflow with parallel specialists. Dependencies: the paper evaluates fixed-topology workflows with bounded retrieval. The relationship may differ for loops, dynamic branching, fan-out graphs, different output lengths, or unbounded histories.
- Provider- and tier-specific memory-cost accounting (cloud cost management and FinOps)
- reducing injected token volume;
- routing the node to a cheaper model;
- reducing output-generation cost;
- changing the workflow structure.
- This prevents teams from incorrectly attributing savings from model routing to memory optimization.
- Dependencies: input and output prices vary across providers and tiers. The paper’s near tier-independent cost-share observation depends on a particular input/output price ratio and should not be generalized without recalculation.
- Operational guardrails for deep or context-heavy agents (customer service, coding, security, and compliance) Production systems can enforce limits such as maximum workflow depth, maximum injected tokens per node, maximum cumulative memory cost, or escalation to summarization when a context budget is exceeded. These controls are especially useful for agents that repeatedly pass intermediate outputs through a chain. Dependencies: hard limits may remove context needed for correctness. Guardrails should be paired with task-specific evaluation, fallback behavior, and human review for high-risk outputs.
- Improved cost attribution for enterprise chargeback and pricing (business operations and finance) SaaS providers and internal AI platforms can allocate costs more accurately to teams, workflows, or customers by reporting memory injection separately from model inference. This is particularly relevant when one department runs shallow agents while another runs deep, context-accumulating workflows. Dependencies: attribution must account for retries, parallel branches, failed calls, caching, durable-store access, and provider-specific billing conventions.
- A reproducible evaluation protocol for agent-memory optimizations (academic research and industrial benchmarking) Researchers can report memory injection tokens alongside accuracy, latency, model tier, workflow depth, and retrieval capacity. The two-pass method provides a framework-independent measurement that can make comparisons between memory strategies more reproducible. Dependencies: exact replication requires reporting tokenizer, provider, prompt-construction rules, cache state, retry semantics, and memory contents. The paper’s benchmark uses synthetic but realistic enterprise data and should not be treated as representative of every domain.
Long-Term Applications
The following applications are promising but require additional experiments, system development, or validation beyond the paper’s evidence.
- Automated cost–accuracy optimization of retrieval capacity (agent compilers and workflow optimizers) A compiler could automatically select a retrieval window for each node or workflow stage by optimizing an objective such as:
Rather than applying one global , the system could retain more context for policy-verification nodes and less for extraction or formatting nodes. Dependencies: this requires a reliable quality signal, broader sweeps across , multiple seeds, and task-specific evaluation. The current paper does not identify the optimal capacity or establish a statistically reliable accuracy effect.
- Memory-aware agent workflow compilation (software engineering and LLM orchestration)
- pruning unnecessary context edges;
- summarizing intermediate outputs;
- passing structured references instead of full text;
- merging compatible nodes;
- reordering tasks to reduce carried context;
- selecting different memory policies for different nodes.
- The paper’s graph-rewriting experiments were approximately cost-neutral in isolation, so these transformations should not currently be assumed to produce savings.
- Dependencies: transformations must preserve task semantics and avoid increasing retries, output length, or model-tier escalation. Static cost reduction does not guarantee end-to-end savings.
- Memory-aware model routing (multi-model serving and cloud inference) A router could jointly consider task difficulty and the cost of the context being carried. For example, a node with a large injected prompt might be routed to a cheaper model if quality permits, or its context might be compressed before being sent to an expensive tier. This would combine the paper’s accounting with existing routing approaches such as FrugalGPT, RouteLLM, or multi-agent routers. Dependencies: the paper holds tier assignment fixed and explicitly finds that tier assignment dominates total cost. Joint routing and memory optimization would require quality predictors, per-tier evaluations, and safeguards against routing context-heavy but accuracy-sensitive tasks to inadequate models.
- Context compression selected by exact injection cost (RAG, coding assistants, and enterprise search)
- extractive summaries of prior agent outputs;
- structured-state representations;
- deduplication of repeated observations;
- prompt-compression models;
- code- or schema-aware compression for SQL, logs, and identifiers.
- Dependencies: compression can alter semantics, especially for structured data, identifiers, numerical values, and policy language. Any deployment needs correctness tests rather than relying only on keyword recall.
- Cache-aware memory architecture (LLM serving infrastructure and cloud platforms)
- first-write costs;
- cache-read savings;
- cache invalidation caused by changing system prompts;
- memory that is reused across multiple downstream nodes.
- Dependencies: prompt caching was not evaluated. Cache effectiveness depends on byte-identical prefixes, minimum cacheable lengths, write/read pricing, prompt layout, and whether generated memory is reused often enough to amortize cache writes.
- TCA optimization for ReAct loops and dynamic graphs (robotics, autonomous software agents, and operations automation)
- browser-use agents;
- robotic task planners;
- incident-response agents;
- autonomous coding and debugging systems;
- financial monitoring agents.
- Dependencies: the paper does not measure loops or dynamic branching. Real systems may have variable iteration counts, tool failures, parallel branches, and changing memory policies, requiring new attribution and forecasting models.
- Billed durable-memory and retrieval-service accounting (cloud databases and enterprise knowledge systems)
- vector-database query charges;
- remote object-store retrieval;
- database egress;
- reranking costs;
- latency penalties converted into operational cost.
- Dependencies: monetary attribution must distinguish token costs from infrastructure costs and avoid double-counting retrieval operations already included in provider or platform bills.
- Standardized benchmarks and observability APIs for agent cost (academia, regulators, and industry standards) The field could adopt a standard reporting schema containing workflow topology, depth, model tier, base tokens, injected tokens, output tokens, cache status, retrieval capacity, retries, latency, and accuracy. Such a schema would enable cross-framework and cross-provider comparisons. Dependencies: providers expose different tokenizer and billing interfaces, and the paper evaluates only one provider/framework combination. Standardization would require agreement on definitions such as “memory,” “inherited context,” and “accumulation.”
- Policy and governance requirements for AI cost transparency (public-sector procurement and regulated industries) Organizations could require vendors to disclose the proportion of LLM expenditure attributable to model inference, base prompts, memory injection, retrieval services, and caching. This would improve procurement decisions and make it easier to detect inefficient or unexpectedly expensive agent designs. Dependencies: cost transparency should not be treated as a substitute for privacy, security, or quality auditing. Reporting memory tokens may also expose information about workflow structure or sensitive retrieved content unless logs are carefully redacted.
- Personal and small-business AI spending controls (daily life and consumer productivity) Consumer agent platforms could expose a simple “context budget” or “memory usage” setting, allowing users to trade continuity against cost. Examples include personal scheduling agents, document assistants, tutoring systems, and home-automation agents that repeatedly carry conversation or task history. Dependencies: consumer pricing may be subscription-based rather than token-based, and users may value continuity more than marginal savings. Automatic truncation or compression must preserve safety-critical instructions and important personal preferences.
Glossary
- Ablation: An experiment that removes or isolates a component to measure its individual effect. “The graph-rewriting transforms are approximately cost-neutral in isolation.”
- Agentic system: A software system in which autonomous or semi-autonomous agents perform tasks through model calls and tool interactions. “We consider an agentic system that answers a query by executing a workflow”
- Application programming interface (API): A programmatic interface that allows software to communicate with an external service. “executed against real model application programming interfaces (APIs).”
- Cache breakpoint: The position in a prompt up to which content must match exactly for a cache entry to be reusable. “a cache hit requires a byte-identical prefix up to the cache breakpoint”
- Capacity-bounded strategy: A memory-management policy that limits the number of retained context entries. “a capacity-bounded strategy retains at most entries”
- Compile-time optimizer: An optimizer that makes decisions before workflow execution begins. “13.6 percent of the variable cost a compile-time optimizer can act on”
- Context accumulation: The progressive growth of prior outputs or other contextual information carried through a workflow. “the mechanism that makes carried context accumulate with depth”
- Context compression: The reduction of a prompt or contextual representation while attempting to preserve its useful information. “A complementary line of work reduces the cost of carried context by compressing it.”
- Context window: The maximum amount of text or tokens that a LLM can process in one input. “paging information between a fast working context and a slower archival store to manage limited context windows”
- Convex growth: Growth whose rate increases as the independent variable increases, commonly represented by a positive quadratic curvature. “the data do not exhibit convex growth at these depths.”
- Critical path: The longest dependency path that determines the depth or completion progression of a directed graph. “For general directed acyclic graphs, is replaced by the maximum fan-in along the critical path.”
- Directed acyclic graph (DAG): A directed graph containing no cycles, often used to represent dependency relationships. “a directed acyclic graph in which each node is a subtask paired with a specialist role”
- Durable store: A persistent storage layer used as a fallback when data are unavailable in a faster memory tier. “the system falls back to the durable store.”
- Execution-grounded evaluation: Evaluation based on whether generated actions or outputs successfully execute in a real environment. “Grading is execution-grounded and deterministic rather than delegated to a model-based rubric”
- Fan-in: The number of upstream inputs or dependencies entering a node. “the maximum fan-in along the critical path.”
- Frontier model: A highly capable, leading-edge LLM, generally more expensive than smaller tiers. “A frontier model is accurate but can cost twenty to twenty-five times what a small model costs per token”
- Graph rewriting: Transformation of a computational graph by changing its nodes, edges, or organization while preserving its intended computation. “the graph-rewriting transforms we implemented are approximately cost-neutral in isolation.”
- Inference cost: The cost associated with the output tokens generated by a model during inference. “$C_{\mathrm{inf}(v)$ is the inference cost, equal to the tokens the node generates times .”
- Instrumentation: The addition of measurements or monitoring mechanisms to software so that its behavior can be recorded. “the instrumentation bears this out”
- Key-value cache: A cache storing attention-related key and value representations to reduce repeated computation for long contexts. “key-value cache compression such as SnapKV”
- Keyword recall: The proportion of required terms or topics that appear in a generated answer. “Keyword recall is coarser than full result matching”
- LangGraph: A framework for constructing graph-structured workflows involving language-model agents. “Agent frameworks including LangGraph~\cite{b14} and AutoGen~\cite{b13} provide infrastructure for multi-agent workflows”
- Memory injection: The insertion of retrieved memory or prior context into a model prompt. “Every agent node also retrieves context from memory before inference and injects it into its prompt.”
- Memory hierarchy: An arrangement of storage layers with different capacities, speeds, persistence properties, or costs. “MemGPT~\cite{b7} introduced a two-tier memory hierarchy for multi-session agents”
- Model tier: A category of models grouped according to capability, price, or performance. “total workflow cost is dominated by model tier assignment”
- Non-billable token count: A token-counting operation that does not incur input or output-token charges. “a two-pass, non-billable token count that measures injected tokens directly”
- Observation-weighted mean: An average in which values are weighted according to the number of observations contributing to each value. “whose observation-weighted mean is 304.4 base tokens per node”
- Orthogonal prior work: Previously established work addressing a separate dimension of a problem rather than the specific mechanism under study. “we hold tier assignment fixed and treat it as orthogonal prior work throughout.”
- Paged key-value cache management: A serving technique that organizes key-value cache memory in pages or blocks to manage long contexts efficiently. “building on paged key-value cache management at the serving layer”
- Prompt caching: Reusing previously processed prompt content to reduce computation or token charges. “Prompt caching is not evaluated; all figures are for the uncached case.”
- Prompt compression: The reduction of prompt length while attempting to retain the information needed for a task. “Task-agnostic prompt compression such as LLMLingua-2~\cite{b18} distills prompts to a fraction of their original token count”
- Prompt assembly: The process of constructing the complete input prompt from system instructions, user content, tools, and retrieved memory. “the two passes are performed by a decorator around the prompt-assembly step”
- Query-level router: A routing mechanism that assigns an entire query to one model based on predicted properties such as difficulty. “RouteLLM~\cite{b2} and Hybrid LLM~\cite{b3} are query-level routers”
- Retrieval-augmented generation (RAG): A method that retrieves external information and supplies it to a LLM as context for generation. “Vector retrieval backends and the broader retrieval-augmented generation literature”
- Retrieval window: The maximum number of memory entries that a node can retrieve and inject into its prompt. “reducing the retrieval window capacity from 32 to 2 entries”
- Self-verification: A procedure in which a model evaluates or checks its own generated output. “AutoMix~\cite{b5} escalates on self-verification failure.”
- Subword tokenizer: A tokenizer that divides text into units smaller than whole words when appropriate. “subword tokenizers split text at boundaries that do not align with words”
- Topological order: An ordering of graph nodes in which every node appears after all of its predecessors. “so execution proceeds in topological order.”
- Total Cost of Agency (TCA): The paper’s decomposition of multi-agent workflow cost into base-prompt, inference, memory-injection, miss-penalty, and accumulation components. “We introduce the Total Cost of Agency (TCA), a decomposition of multi-agent workflow cost”
- Token attribution: The assignment of token counts or token-related costs to particular components of a workflow. “Token attribution is exact via the two-pass count of Section~\ref{sec:method}.”
- Tokenization: The process of converting text into tokens used by a LLM. “Both are tokenized, and the difference is attributed to injection”
- Warm tier: A fast-access memory layer that stores recently or frequently used context. “memory is retrieved from a warm tier of capacity $K_{\mathrm{default}$”
- Workflow depth: The maximum node depth in a workflow, where node depth is determined by the longest path from a source node. “its share rises from a structural zero at workflow depth one to 27.6 percent at depth six.”
