You don't need long context
Abstract: Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier LLMs sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context LLMs.
Summary
- The paper demonstrates that end-of-conversation compaction and selective reading maintain task performance.
- Using compact and strategically managed contexts reduces inference cost by up to $200×$ for selective-reading configurations, demonstrating significant practical benefits.
- Single-action compaction proves the idea of semantic state preservation concept enabling more sustainable performance and even surpassing baseline accomplishments in simplified agent models.
Research question and thesis
“You don't need long context” (2611.05306) examines whether long-context LLMs are necessary for two workload classes that commonly generate large prompts: long-horizon agentic tasks and long-input information-processing tasks. It further asks whether long-horizon agents can operate with no more than one environment action per model context. The paper’s central claim is affirmative across all three questions: frequent context compaction and selective input access preserve task performance while substantially reducing peak context length, inference cost, and latency.
The argument is motivated by the deployment economics of autoregressive decoding. With a KV cache already populated, each generated token must attend over the existing context, so decode latency, memory consumption, and serving cost increase with context length. The authors’ measurements on open-weight models indicate that shrinking the configured context window from $256$K to $16$K tokens increases sustained decode throughput by for GLM 5.2, for Kimi K2.6, and for Qwen 3 Coder Next. These figures are not merely hardware-efficiency measurements: they establish that context management can yield deployment benefits comparable to, or greater than, aggressive quantization and sparsity techniques.
Figure 1: Estimated cost of generating one million output tokens increases substantially with the configured context window.
The paper distinguishes a model’s available input data from the subset that must be present in its active context at a particular inference step. This distinction is crucial. A long software-engineering trajectory may contain thousands of commands and tool outputs, but only a small state representation may be required for the next action. Similarly, a multi-document question may be accompanied by hundreds of thousands of tokens while requiring evidence from only a few localized passages. The paper therefore treats context as an actively managed computational state rather than as a passive transcript.
Computational motivation
The systems measurements establish three related costs of long context. First, later tokens are slower to decode because the active sequence is longer. Second, the KV cache occupies more memory as the context grows. Third, under fixed GPU allocation, lower decode throughput translates directly into higher cost per generated token. Local experiments on an NVIDIA DGX Spark show that the inter-token latency of several open-weight models approximately doubles over the tested context range, while memory usage also rises.
Figure 2: Decode latency and memory footprint increase with input context length across locally deployed models.
The cloud measurements use vLLM deployments on H100 GPUs, with prefix caching and saturated concurrency. They measure post-prefill decode throughput rather than prompt-processing throughput, isolating the cost most directly affected by context depth during generation. The resulting throughput curves collapse as the configured context window increases.
Figure 3: Sustained decode throughput falls sharply as the configured context window grows.
The authors convert these measurements into estimated retail-cost ratios for frontier closed-weight models. This extrapolation is an important methodological assumption: the cost analysis applies a context-dependent factor inferred from three open-weight deployments and combines it with published model prices. The reported savings should therefore be interpreted as modeled deployment costs rather than direct measurements of the closed-weight providers’ internal serving systems.
Context-management methodology
The evaluation uses a common agent loop in which the model receives a task, issues a Bash action, observes the result, and continues. Three context regimes are compared:
- Full-History Accumulation (FHA): every action and observation remains in one conversation.
- End-of-Conversation Compaction (EoCC): once a threshold is reached, the model produces a bounded task-state recap, the conversation is restarted, and execution continues from that recap.
- Single-Action Compaction (SAC): each context performs one environment action, after which the conversation is discarded and replaced by a new context containing the task and updated semantic state.
Compaction is deliberately not treated as generic summarization. The authors report that effective compactions preserve the task specification, verified facts, current progress, changed artifacts, validation results, failed approaches, and the next action. They characterize the desired output as a forward-looking execution brief rather than a narrative summary. This design choice is consequential: a generic summary may overrepresent repeated or historically salient material while omitting exact paths, values, error messages, or unresolved obligations that are required for subsequent execution.
For long-input tasks, the complete source material is placed in the agent’s filesystem rather than in the initial prompt. The agent uses bounded commands such as grep, sed, and custom document-reading utilities to retrieve only relevant excerpts. This converts a long-input problem into a sequence of selective-reading actions, with compaction available if the retrieved evidence itself becomes large.
Long-horizon task results
The long-horizon evaluation covers SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.1, FRAMES, -bench, and BFCLv3. The principal models are Claude Opus 4.5 and GPT 5.6 Sol. Across the evaluated compaction thresholds, accuracy generally remains close to the FHA baseline, while estimated optimized retail cost falls substantially.
For Claude Opus 4.5, relative accuracy across the long-horizon configurations ranges from to percentage points; for GPT 5.6 Sol, it ranges from to points. The aggregate range obscures the more relevant observation that no more than five of the 22 threshold configurations for either model lose more than two percentage points relative to baseline. Estimated cost savings range from to for Opus 4.5 and from to for GPT 5.6 Sol.
Figure 4: Long-horizon accuracy remains broadly stable as context limits decrease, while modeled optimized cost falls.
The software-engineering results illustrate both the strength and the qualification of the claim. On SWE-bench Verified, Opus 4.5 scores 70.0% under FHA and 74.8% under EoCC at a $24$K total-context limit with an $18$K compaction threshold. GPT 5.6 Sol rises from 84.4% to 88.8% under a $48$K/$36$K configuration. The tightest $16$K/$12$K setting reduces Opus accuracy to 68.8% but leaves GPT accuracy at 84.6%. On Terminal-Bench, Opus improves from 47.2% to 52.8%, while GPT reaches 88.8% compared with an 87.6% baseline.
The result is not uniform across task types. On FRAMES, compaction reduces accuracy by 1.3–3.7 points for Opus and 1.5–3.3 points for GPT, although estimated optimized cost falls by and , respectively, at the $16$K/$12$K setting. This establishes a practical trade-off: short contexts are usually viable, but information-dense multi-hop retrieval can incur measurable losses when the state budget becomes too restrictive.
The open-weight results provide a more substantial counterexample to the paper’s broad thesis. Robustness varies considerably across architectures and post-training recipes. Qwen 3 Coder Next falls from 53.0% to 3.6% on SWE-bench Verified at $16$K/$12$K, apparently because verbose trajectories leave insufficient room for useful progress between compactions. Gemma 4 31B falls from 65.2% to 58.0%, and Step 3.7 Flash from 70.0% to 56.6%. By contrast, Qwen 3.6 27B declines only from 68.6% to 65.0%, while MiniMax M2.7 declines from 72.6% to 67.8%.
Thus, the evidence supports the weaker and more defensible proposition that many long-horizon tasks can be solved with short contexts, not that arbitrary models are insensitive to aggressive compaction. Model-specific trajectory verbosity, state-tracking ability, and post-training clearly affect the operating point.
Long-input tasks and selective reading
The long-input experiments evaluate LongBench v2, LOONG, AA-LCR, Needle in a Haystack, RULER, Counting-Stars, and BABILong. The baseline places the full input in the model context; the selective-reading condition exposes documents through filesystem tools and instructs the model to search for and verify relevant passages.
The main result is that selective reading frequently improves both accuracy and cost. On LongBench v2, Opus remains within 0.4 points of its 68.0% baseline at a $16$K total context limit, while GPT improves from 70.8% to 72.6%. On LOONG, every selective-reading configuration improves both average score and perfect-answer rate for both models. At $16$K/$12$K, Opus reaches 79.3% average and 53.4% perfect answers, compared with 70.1% and 38.9% for the full-input baseline; GPT reaches 82.7% and 51.9%, compared with 79.3% and 44.4%.
Figure 5: Selective reading and compaction maintain or improve long-input accuracy while reducing modeled cost.
AA-LCR provides a particularly strong result for Opus: at $16$K/$12$K, accuracy rises from 65.0% with full-input processing to 84.0% with selective reading. GPT changes from 82.0% to 83.0%. On RULER, mean recall increases from 62.5% to 95.7% for Opus and from 65.5% to 96.4% for GPT, with peak live context bounded at $16$K. Counting-Stars reaches 99.7% mean score for both models and improves exact match to 81.2%.
BABILong yields another large difference: Opus improves from 67.1% to 92.0%, and GPT from 70.7% to 94.0%, under a $16$K/$12$K limit. These gains do not demonstrate that short context intrinsically improves reasoning. They instead indicate that full-input ingestion can be a poor interface: selective retrieval may reduce distraction, expose relevant evidence in a more usable order, and avoid the degradation associated with processing very long irrelevant prefixes.
The paper appropriately qualifies its strongest retrieval results. Needle-in-a-Haystack allows lexical overlap between the query and inserted target, making search comparatively straightforward. The 100% selective-reading accuracy therefore does not establish equivalent performance for adversarially disguised evidence or queries requiring broad semantic discovery.
The relationship between document size and active conversation length is central to the paper’s interpretation. In both LongBench v2 and LOONG, conversation lengths remain concentrated in relatively narrow bands while document lengths span orders of magnitude. This supports the claim that the amount of evidence required for an individual question is often much smaller than the total source corpus.
Figure 6: LongBench v2 conversation lengths remain comparatively bounded as document length increases.
Figure 7: LOONG shows a similar separation between total document length and active conversation length.
Single-action compaction and semantic state
SAC tests the strongest operational version of the thesis: one action per context, with only a compact semantic state transferred between model invocations. For Opus 4.5, SAC preserves baseline SWE-bench Verified accuracy almost exactly, reaching 70.2% versus 70.0%, and improves Terminal-Bench from 47.2% to 51.8%. GPT 5.6 Sol reaches 82.4% on SWE-bench Verified and 84.3% on Terminal-Bench, reductions of 2.0 and 3.3 points from its FHA baselines.
The context reduction is substantial. Median peak live context under SAC is only 7–9K tokens across the reported model–benchmark combinations. Maximum peak context falls from 130–200K under FHA to 15–37K under SAC. These results establish that the model need not preserve a multi-turn transcript in its active context to perform sequential software-engineering work, although the modest GPT degradation demonstrates that semantic-state transfer is not lossless.
The authors use this observation to propose Semantic State Space Models (SSSMs). An SSSM would explicitly maintain a natural-language state between inference instances. Each instance would interpret the state, select and execute an action, observe the result, and update the state for the next instance. Unlike a conventional neural state-space model, the state is intended to remain inspectable and semantically interpretable.
The proposal is conceptual rather than an experimentally trained architecture. The experiments show that existing frontier models can approximate SSSM behavior through prompting and harness-level compaction; they do not establish that dedicated SSSM post-training is more data-efficient, more accurate, or more robust than full-history training. The paper’s claim that such models could match full-history agents therefore remains an open empirical question.
Limitations and open questions
The strongest limitation is benchmark and harness dependence. The study evaluates two closed-weight frontier models in depth, but the open-weight results show large variation, including catastrophic degradation for Qwen 3 Coder Next under tight compaction. The broad conclusion cannot therefore be transferred automatically across model families, context-training procedures, or agent frameworks.
The cost analysis also depends on an extrapolation from measured open-weight throughput. It assumes a common context-dependent scaling factor, anchors it to retail prices, and estimates optimized costs for proprietary models. These estimates are useful for comparing regimes but should not be read as provider-specific accounting results.
Selective reading changes the task interface. The baseline receives the complete input directly, whereas the proposed system receives documents through search tools and detailed instructions. This is a legitimate systems comparison, because the paper’s thesis concerns workload design rather than fixed prompt formatting, but it conflates context reduction with tool-mediated retrieval. Performance may depend on filename conventions, searchable text, command availability, query formulation, and the presence of lexical cues.
The paper also does not establish worst-case guarantees. Its coin-toss example correctly shows that lossy compaction can destroy information required later. Selective access can mitigate this only if the complete history is stored, searchable, and recoverable through an adequate interface. Tasks requiring exact global aggregation, high-precision temporal dependencies, or repeated access to numerous weakly indexed facts may stress this assumption.
Finally, the experiments focus on accuracy and modeled cost more than on state fidelity, compaction-call overhead, failure recovery, or human steerability. The reported compaction prompts preserve information that is useful for autonomous execution, but human-in-the-loop workflows may require preserving user preferences, rationale, and alternative plans that are not necessary for benchmark completion.
Conclusion
“You don't need long context” (2611.05306) presents a systematic systems argument that long input size and long interaction horizon do not, by themselves, imply a need for long active contexts. Across long-horizon and long-input benchmarks, end-of-conversation compaction and selective reading usually preserve baseline accuracy while producing modeled cost reductions of up to and, for some long-input configurations, more than . Single-action compaction further shows that frontier models can maintain useful sequential behavior with only a compact semantic state between actions.
The evidence supports aggressive context management as a practical deployment strategy, but not the universal elimination of long-context capability. Model-specific failures, task-dependent information requirements, retrieval-interface assumptions, and extrapolated cost models remain material qualifications. The paper’s most consequential open question is whether post-training explicitly optimized for semantic state maintenance can provide the robustness of full-history agents without their computational overhead.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
1. What is this paper about?
The paper asks a simple but important question:
Do AI LLMs really need to remember huge amounts of text at once?
Modern LLMs can sometimes handle hundreds of thousands or even millions of “tokens.” Tokens are small pieces of text, such as words or parts of words. However, using very large contexts makes AI systems:
- slower,
- more expensive to run, and
- more demanding on computer memory.
The researchers argue that AI models may work almost as well if they use a much shorter working memory and regularly summarize or retrieve information when needed.
Their main message is:
AI systems often do not need to keep their entire history or every document in memory at the same time.
2. What questions did the researchers study?
The paper focuses on three main research questions:
- Can AI complete long tasks without keeping a very long conversation history? For example, can an AI programmer fix a complicated software problem without remembering every command it has used?
- Can AI answer questions about very large documents without reading all of them at once? For example, can it find the important information in many books, reports, or research papers by looking only at useful sections?
- Can AI complete a task while taking only one action per conversation? After each action, the AI would summarize what it learned, forget the old conversation, and start a fresh one.
The researchers wanted to discover whether long context windows are truly necessary, or whether they are mostly convenient but inefficient.
3. How did the researchers test this?
Comparing different memory strategies
The researchers tested AI systems using three different ways of handling information.
Full-History Accumulation (FHA) is the usual method. The AI keeps adding every message, action, tool result, and previous decision to one large conversation.
This is like a student keeping every single piece of paper from an entire school year on their desk.
End-of-Conversation Compaction (EoCC) allows the conversation to grow for a while. When it becomes too large, the AI writes a shorter summary of the important information. The old conversation is then replaced by the summary.
This is like cleaning the desk by putting the important notes into one organized folder.
Single-Action Compaction (SAC) is even more extreme. The AI receives the task and its current summary, performs one action, sees the result, updates the summary, and then starts a new conversation.
This is like solving a mystery one clue at a time while carrying only a small notebook.
Letting AI read large inputs selectively
For tasks involving huge documents, the researchers did not always place the complete documents inside the AI’s initial conversation. Instead, they stored the documents as files.
The AI could then use simple computer commands to read only the parts it wanted. For example, it might:
- search for a keyword,
- read a particular section,
- inspect the beginning or end of a file, or
- compare information from several selected documents.
This is similar to using a library catalog and reading only the pages that seem relevant instead of carrying every book around at once.
The tests they used
The researchers tested the systems on many tasks, including:
- fixing real software problems,
- completing command-line computer tasks,
- searching through articles,
- answering questions using multiple documents,
- using tools while talking with a simulated customer, and
- finding important facts hidden inside very long texts.
They used powerful LLMs, including closed models such as Claude Opus 4.5 and GPT 5.6 Sol, as well as several open models for speed and cost measurements.
4. What did they find?
Shorter contexts usually kept accuracy about the same
For long, multi-step tasks, regularly summarizing the conversation usually caused little or no loss in performance.
Across the tested long-horizon tasks, the researchers found that accuracy was generally close to the full-history approach. In some cases, the summarized version even performed better.
For example, the difference from the normal full-history method was usually only a few percentage points. This suggests that the AI did not need to remember every previous message in exact detail.
Selective reading worked well for huge documents
When the AI could search and read large documents in small pieces, it usually performed as well as, or better than, a system that received the entire document at once.
This is important because a document may contain hundreds of thousands or millions of tokens, while the answer may depend on only a few paragraphs.
The AI did not need to hold the entire document in its active memory. It could look up information when necessary.
One action per context was surprisingly effective
The single-action method kept the active context extremely short. In the experiments, the typical active context was often only about 7,000 to 9,000 tokens, compared with much larger full conversations.
Despite this, the systems still completed many software and terminal tasks successfully.
Some results improved, while others became slightly worse. For example, one model lost a few percentage points on certain tasks. However, the researchers did not observe a complete collapse in performance.
Short contexts saved a lot of money and time
The paper reports that long contexts can make AI generation much more expensive.
For some open models, reducing the context size from 256,000 tokens to 16,000 tokens increased generation speed by about:
- 3.6 times for one model,
- 5.6 times for another, and
- 7 times for a third.
The estimated cost savings were also large:
- roughly 2 to 9 times cheaper for many long-horizon tasks, and
- sometimes more than 50 or even 200 times cheaper for certain long-input tasks.
The exact savings depend on the model and task, but the general idea is clear: keeping less text active can greatly reduce computing costs.
Important information can sometimes be lost
The researchers also point out a limitation. Summarizing is not perfect. Some details may disappear.
They give an example involving repeated coin tosses. If an AI must remember the exact result of every toss, a short summary might accidentally leave out some results. The AI could then fail if it later needs one of the missing details.
However, the researchers argue that this problem can often be solved by keeping the old information in a searchable file or database. The AI can retrieve it later when needed instead of keeping everything in its immediate conversation.
5. Why are these findings important?
The results challenge the common belief that bigger context windows are always better.
A huge context window is useful because it gives the AI the ability to see a lot of information. But simply having access to information does not mean the AI must keep all of it in active memory all the time.
The paper suggests a better design:
- Keep a short summary of the current situation.
- Let the AI take an action.
- Save important information outside the conversation.
- Retrieve older information only when necessary.
- Repeat this process as the task continues.
This could make AI systems:
- cheaper to operate,
- faster to respond,
- easier to run on personal computers,
- more practical for private or offline use, and
- less dependent on extremely large and expensive hardware.
The authors call a possible future system based on this idea a semantic state space model. In simple terms, this would be an AI that carries a compact description of what matters instead of dragging along its entire conversation history.
Conclusion
This paper argues that AI models do not always need enormous memories. For many long tasks, they can work effectively by keeping a short, meaningful summary and looking up older information only when needed.
The findings do not mean that long contexts are never useful. Some tasks genuinely require exact details from a long history. However, the research suggests that many current AI systems are wasting computing power by carrying around far more information than they actively need.
The potential impact is significant: future AI assistants could become faster, cheaper, and more efficient while still solving complicated problems.
Practical Applications
Immediate Applications
The paper’s results support deploying short-context agent architectures today, especially where workflows involve repeated tool calls, long documents, or privacy-sensitive local inference. The main mechanisms are end-of-conversation compaction, single-action compaction, and selective input reading.
- Lower-cost cloud LLM serving for agentic workloads — software, customer support, and enterprise automation.
- monitors context length;
- triggers compaction at a configurable threshold;
- retains the original task, current objectives, open subproblems, decisions, and unresolved errors;
- starts a fresh inference context with the compacted state.
- Dependencies: Compaction quality must be reliable; thresholds should be calibrated by task type; applications requiring exact historical recall may need an external log or retrieval tool.
- Efficient coding agents for repository maintenance — software engineering and DevOps.
- IDE coding assistants with bounded context;
- CI agents that inspect, patch, test, and revise repositories;
- terminal agents operating in sandboxed containers;
- pull-request repair bots.
- Dependencies: The agent must be able to reconstruct the current repository state from files, tests, version-control history, and a compact task state. Performance may decline modestly for models or tasks that depend heavily on conversational continuity.
- Selective-reading document assistants — legal, finance, research, and compliance.
- multi-document financial analysis;
- legal-case comparison;
- regulatory consultation review;
- literature synthesis;
- audit and compliance investigations.
- Dependencies: Retrieval and navigation tools must expose enough structure for the model to find relevant evidence. The system should preserve citations, document identifiers, and page or section locations to reduce unsupported conclusions.
- Reduced latency and improved throughput for LLM APIs — cloud infrastructure.
- context-aware routing;
- separate short-context and long-context model pools;
- automatic history summarization;
- pricing based on actual context-related compute rather than a uniform token rate.
- Dependencies: Results depend on model architecture, batching, KV-cache implementation, quantization, hardware, and request distribution. Shortening the configured window should not be treated as universally equivalent to improving task performance.
- On-device and privacy-preserving assistants — local software, healthcare administration, and industrial operations.
- offline coding assistants;
- local document classification;
- private clinical-administrative search;
- manufacturing troubleshooting;
- field-service copilots.
- Dependencies: The local model must fit available memory and provide sufficiently accurate tool use. Sensitive applications require access control, audit logs, encryption, and safeguards against the model reading unauthorized files.
- More predictable context budgeting in agent platforms — software infrastructure.
- live context length;
- total tokens across all inference calls;
- compaction frequency;
- external-memory retrieval;
- maximum action count.
- This would allow developers to optimize for latency, cost, and reliability rather than simply selecting the largest available context window.
- Dependencies: Total token consumption can rise if compaction causes repeated work, so evaluation must measure both peak context and cumulative inference cost.
- Teaching efficient agent design in academia and engineering education. Agent-development courses and research prototypes can use the paper’s harnesses as a baseline for comparing full-history, end-of-conversation, and single-action regimes. Students can measure accuracy, peak context, total token usage, latency, and cost on coding, retrieval, and tool-use tasks. Dependencies: Benchmark results should not be generalized beyond the tested models and workloads without additional experiments.
- Everyday long-document assistance — personal productivity and education.
- question answering over course materials;
- exam revision;
- instruction-manual troubleshooting;
- personal knowledge management;
- summarization of large email or note collections.
- Dependencies: Users need transparent citations and the ability to inspect retrieved passages. Selective reading may miss relevant information when the task requires exhaustive coverage rather than targeted reasoning.
Long-Term Applications
These applications require further validation, specialized training, improved memory systems, or deployment at larger scale. The paper demonstrates feasibility, but not yet universal robustness.
- Semantic state space models for long-running autonomous agents — robotics, software, and operations.
The proposed semantic state space model idea could replace growing conversational histories with a persistent, structured state containing goals, world facts, plans, constraints, artifacts, and uncertainty. A robot or enterprise agent would:
- observe the environment;
- perform one action;
- update its semantic state;
- begin the next inference with only the relevant state and observations. Potential systems include warehouse robots, laboratory automation, autonomous IT operations, and multi-day research agents. Dependencies: The state representation must avoid losing safety-critical details, support contradictions and revisions, and distinguish durable facts from temporary assumptions. Formal verification and recovery mechanisms are likely necessary.
External episodic memory for exact historical recall — healthcare, finance, law, and safety-critical systems.
- immutable event logs;
- structured databases;
- vector and keyword retrieval;
- temporal and provenance metadata;
- model-generated semantic summaries.
- This would allow an agent to maintain a short working context while recovering exact past events when needed, such as a patient’s prior instruction, a financial transaction, or a robot’s sensor event.
- Dependencies: Retrieval must achieve high recall, records must be tamper-evident, and systems must comply with privacy, retention, and audit requirements. Exact lookup should not rely solely on approximate vector search.
- Training models specifically for short-context, tool-mediated reasoning — model development and academia.
- accurate state compression;
- explicit uncertainty;
- evidence citation;
- efficient retrieval;
- avoidance of redundant actions;
- preservation of task-critical invariants.
- This could produce models that are natively efficient rather than relying on generic models to summarize their own history.
- Dependencies: Training datasets must include failures caused by omitted context, stale summaries, conflicting evidence, and retrieval errors. Improvements must be tested on realistic, adversarial, and domain-specific tasks.
- Context-adaptive inference systems — cloud platforms and enterprise AI.
- a short direct context;
- selective retrieval;
- compaction;
- multiple specialized agents;
- a full long-context pass.
- The platform could allocate GPU memory and model capacity according to estimated information requirements, reducing unnecessary long-context computation.
- Dependencies: Reliable workload classification is required. Incorrectly classifying a task as short-context could cause silent omissions, while overly conservative routing would erase the efficiency gains.
- Long-horizon multi-agent collaboration with shared semantic state — robotics, research, and business processes. Multiple agents could communicate through a compact shared state rather than forwarding complete transcripts. For example, a planning agent, execution agent, verifier, and documentation agent could operate on the same structured task state while retaining full interaction logs externally. Dependencies: Shared-state schemas, version control, conflict resolution, permission boundaries, and provenance tracking are essential. Without them, compaction could propagate an early agent’s error across the entire system.
- Regulatory and policy standards for context-efficient AI.
- context-window configurations;
- compaction and retrieval behavior;
- information-loss risks;
- retention of original evidence;
- cost and energy impacts of deployment choices.
- Public-sector procurement could favor systems that achieve comparable accuracy with lower compute and memory requirements.
- Dependencies: Policymakers need independent evaluations across domains. Efficiency should not be rewarded at the expense of transparency, accessibility, reliability, or required record retention.
- Energy-efficient large-model deployment — data centers and edge computing. Because shorter contexts improve throughput and reduce memory pressure, large-scale deployment could use context compaction as an energy-efficiency measure. This is particularly relevant for high-volume assistants, code-generation services, and edge devices with constrained memory. Dependencies: The paper measures throughput and estimated monetary cost, not complete energy consumption or lifecycle emissions. Energy benefits should therefore be validated using hardware-level power measurements and realistic workload mixes.
- Formal memory guarantees for autonomous agents — safety-critical robotics and regulated automation. Future systems could attach guarantees to semantic state updates, such as ensuring that safety constraints, user permissions, emergency conditions, and irreversible decisions are never discarded during compaction. This might involve typed memory, invariant checking, and human approval for state transitions. Dependencies: Such guarantees require domain-specific formalizations of “task-critical information” and reliable detection of when a task exceeds the assumptions under which short-context operation is safe.
Glossary
- Agentic workflow: A process in which a LLM autonomously performs multiple actions, observes results, and continues toward a goal. “In agentic workflows, it typically coincides with one agent turn of a conversation.”
- Attention mechanism: A neural-network operation that determines which tokens should influence the representation of another token. “linear-cost attention-based mechanisms remain the backbone of all popular LLMs.”
- BF16 (bfloat16): A 16-bit floating-point numerical format commonly used for neural-network computation. “Qwen 3 Coder Next is served in BF16.”
- Compaction: The lossy compression of conversational history into a shorter, task-relevant representation. “Compaction is a lossy context compression method.”
- Context window: The maximum token space that a LLM can process in one invocation. “The context window is the space of tokens an LM can comprehend at any given point.”
- Decode throughput: The rate at which a serving system generates output tokens during decoding. “We report the highest-throughput setting with no KV-cache preemption.”
- Dense model: A model in which all parameters are generally active for each input. “Qwen 3.6 27B is a 27B-parameter dense model served in [ADDRESS].”
- End-of-Conversation Compaction (EoCC): A strategy that summarizes the conversation when it reaches a specified length threshold and then starts a fresh conversation. “This regime follows the full-history [PERSON_NAME] loop until the length of the conversational context crosses a pre-specified compaction threshold.”
- FP8: An 8-bit floating-point representation used to reduce memory use and computational cost. “[PERSON_NAME] additionally with an FP8 KV cache”
- Full grouped-query attention: An attention architecture that uses grouped query attention throughout the model rather than only in selected layers. “It uses full-softmax grouped-query attention throughout”
- Full-History Accumulation (FHA): A context-management regime that retains every model action and environment observation throughout a task. “The baseline full-history accumulation regime appends every model action and environment observation to a single conversation for the duration of the task.”
- Gated grouped-query attention: A grouped-query attention mechanism whose information flow is controlled by learned gates. “uses head-wise gated grouped-query attention”
- Global attention: An attention mechanism in which a layer can attend across the entire available context rather than only a local window. “It interleaves $1024$-token local sliding-window attention with global-attention layers”
- Grouped-query attention: An attention design in which multiple query heads share key and value heads to reduce computational and memory costs. “conventional full grouped-query [PERSON_NAME] attention”
- Harness: Software infrastructure that runs an agent, supplies its environment, and manages its interaction with a LLM. “We use two closely related base harnesses.”
- Inference step: A single invocation of a LLM to generate output. “An inference step is one invocation of an LM to produce output.”
- Inter-token latency: The time interval between the generation of consecutive output tokens. “For several models, later tokens take approximately 2x longer to generate”
- KV cache: Stored key and value representations from previous tokens that are reused during autoregressive generation. “Input request lengths are drawn uniformly at random.”
- Long-horizon task: A task requiring many sequential model actions whose later actions depend on earlier observations or decisions. “A long-horizon task is a task whose completion requires many sequential model actions”
- Long-input task: A task accompanied by a large amount of input information that would form a long context if inserted directly. “A long input task is a task which is accompanied by a long input”
- MoE (mixture-of-experts) model: A model containing multiple expert subnetworks, only some of which are activated for each input. “Qwen 3 Coder Next is a sparse MoE model with 80B total and 3B active parameters”
- Needle in a Haystack (NIAH): A benchmark that tests whether a model can retrieve a target fact embedded in a large quantity of irrelevant text. “Needle in a Haystack (NIAH) consists of target sentences inserted at controlled depths in long bodies of unrelated essay text”
- On-device inference: Running a model locally on the device where the task data is stored, rather than on a remote service. “Such deployments are relevant when source code or other task data must remain on-device for privacy or cost reasons.”
- Post-training: Model optimization performed after initial pretraining, often using supervised or preference-based data. “we can likely preserve them by making the appropriate post-training adjustments to future models.”
- Prefix caching: Reusing computations for a shared initial sequence of tokens across multiple requests. “We prefill the longest prefix once, enable prefix caching”
- Quantization: Representing model weights or activations with lower-precision numerical formats to reduce memory and computation. “Note that the extent of cost reductions ( to ) rivals or exceeds that of many other orthogonal optimization methods such as aggressive quantization”
- Selective input reading: Providing long documents through an environment so that an agent retrieves only relevant portions when needed. “We enable selective input reading by storing each long input as one or more documents”
- Semantic state: A compact representation of the information needed to continue a task across separate model contexts. “This regime maintains a compact semantic state and essentially no conversational history.”
- Serving configuration: The deployment settings that determine how a model is hosted and what limits apply to its operation. “The context window size is the maximum number of tokens that an LM endpoint can process in one invocation, whether imposed by the model's context window or narrowed by its serving configuration.”
- Single-Action Compaction (SAC): A regime that creates a fresh context after exactly one environment action while carrying forward a compact state. “Each conversation receives the task and current semantic state, performs exactly one environment action, observes its result, and updates the semantic state.”
- Sliding-window attention: An attention mechanism that restricts each token’s attention to a fixed local region. “It interleaves $1024$-token local sliding-window attention with global-attention layers”
- Sparse model: A model designed so that only a subset of its parameters or connections is active for a given input. “Step 3.7 Flash (198B A11B) is a sparse MoE model”
- Structured sparsification: Reducing model computation by systematically removing structured groups of parameters or connections. “aggressive quantization or structured sparsification”
- Throughput: The quantity of output work completed per unit of time, here measured in generated tokens per second. “we measure sustained aggregate decode throughput as completed output tokens divided by elapsed time.”
- Tokenization: The process of dividing text into tokens for language-model processing. “Word-token conversions use approximately $1.33$ tokens per English word”
- Tool call: A model-generated request to invoke an external program, command, or service. “It may include a thinking trace, tool calls, user-oriented outputs, or other model output artifacts.”
- Unified memory: A memory architecture in which CPU and accelerator components share a common memory pool. “whose 128[ADDRESS] of unified memory leaves about 109[ADDRESS] available for local LM serving”
- Zero-shot retrieval: Retrieving information from a context without relying on an explicit lexical match or task-specific training example. “They do, however, require aggregation, variable tracking, or reasoning over multiple retrieved facts”