Papers
Topics
Authors
Recent
Search
2000 character limit reached

Benchmarks Are Surprisingly Compaction-Friendly

Published 22 Oct 2026 in cs.LG and cs.AI | (2610.31778v1)

Abstract: Long horizon agentic tasks are now pushing LLM agents past their context window. To continue the task, the agent scaffold has to free up the context window, a process known as context compaction. Compaction is generally lossy by design: agents can silently lose task constraints, instructions, or their reasoning behind earlier decisions. Although compaction is a standard component of coding agent scaffolds, it has not been studied systematically with frontier models on realistic, long horizon agentic tasks. Previous work evaluate smaller open weight models on toy or synthetic trajectories. We fix the agent scaffold (OpenCode), implement five compaction strategies inside it, and evaluate three frontier models on three long horizon benchmarks (PaperBench, ProgramBench, PostTrainBench). For the two 1M context models, the compaction threshold does not measurably affect the task performance, whereas GPT-5.3-Codex, the model with the smallest context window, performs worse at lower thresholds on PaperBench and, less clearly, on PostTrainBench. No compaction method outperforms default self-summarization, and methods that discard more content cost some models up to 20 points while leaving other models the same. Compaction can, however, silently remove task constraints: on ProgramBench, the first compaction summary already drops about one third of the task's anticheating rules on average, and this rises monotonically with the number of compactions. On real user-agent sessions from SWE-chat, instructions about how the agent should work and what it should not do are dropped around 6 times more than specifications of what to build. We argue that current benchmarks are compaction-friendly: i.e., the objective is fixed and the agent's progress stays on disk, so most of what compaction discards can be recovered, except constraints that don't exist anywhere else.

Summary

  • The paper finds that while context compaction generally does not significantly impact task scores, it systematically removes key constraints and instructions, even when prompted to preserve them.
  • Task thresholds were analyzed and threshold effects evaluated for 1M and smaller token models with varying benchmark scores
  • default self-summarization was tested against alternative strategies, revealing no consistent performance improvement and highlighting model-specific results.

Context compaction is a necessary mechanism for long-horizon LLM agents whose trajectories exceed the model context window. It is also intrinsically lossy: the scaffold replaces the interaction history with a shorter representation, usually generated by the agent itself. “Benchmarks Are Surprisingly Compaction-Friendly” studies whether this loss affects downstream task performance and whether it compromises adherence to task constraints (2610.31778).

The paper’s central claim is deliberately bifurcated. On the benchmarks evaluated, compaction usually has little effect on measured task performance, particularly for models with 1M-token context windows. At the same time, compaction systematically removes certain classes of instructions, especially process constraints and prohibitions, even when the summarization prompt explicitly asks the model to preserve them. The apparent robustness of benchmark scores therefore coexists with a substantial failure of constraint preservation.

Research questions and experimental design

The study addresses four questions: whether the compaction threshold affects performance, whether the compaction strategy matters, whether compaction erodes task constraints, and whether existing benchmarks are structurally favorable to compaction.

The authors fix OpenCode as the agent scaffold to isolate compaction from scaffold-level differences in system prompts, tool handling, retry logic, and context management. They evaluate four frontier models:

  • Claude Opus 4.6, with a 1M-token context window;
  • GPT-5.4, with a 1.05M-token context window;
  • GPT-5.3-Codex, with a 400K-token context window;
  • Gemini 3.1 Pro Preview, with a 1M-token context window.

The principal experiments use PaperBench, ProgramBench, and PostTrainBench. PaperBench evaluates research-paper reproduction; ProgramBench evaluates black-box software reimplementation under anti-cheating constraints; and PostTrainBench evaluates autonomous post-training of small LLMs under a compute budget. These benchmarks generate long trajectories and expose agents to realistic tool use, file manipulation, iterative debugging, and experiment management.

The compaction threshold is varied from 160K to 380K tokens. Five strategies are compared:

  1. default self-summarization;
  2. tool-output pruning combined with self-summarization;
  3. filesystem offloading of the full pre-compaction history;
  4. removal of prior reasoning blocks before summarization;
  5. summarization by a smaller model, Claude Haiku 4.5.

Task performance is measured using the native benchmark metrics. ProgramBench receives additional evaluation of constraint preservation: at every compaction, an LLM judge determines whether the summary still contains each of three anti-cheating categories. The study also analyzes 269 user constraints extracted from 44 real SWE-chat sessions containing at least one compaction event (Baumann et al., 22 Apr 2026).

Threshold effects are weak for 1M-context models

For Claude Opus 4.6 and GPT-5.4, task performance remains statistically similar as the compaction threshold varies between 160K and 380K tokens. This is a strong result because the thresholds induce materially different numbers of compaction events: at 160K, GPT-5.4 averages approximately 15 compactions per run, whereas at 380K it averages fewer than two. Despite this difference, neither model exhibits a meaningful degradation across the threshold sweep.

The result does not establish that compaction frequency is irrelevant in general. The tested thresholds cover only 16–38% of the 1M-token context windows, rather than the 90–95% utilization levels commonly used by production scaffolds. The experiment therefore evaluates relatively early compaction within a large available context, not compaction near the actual maximum window.

GPT-5.3-Codex behaves differently. On PaperBench, its accuracy increases from approximately 0.38 at a 160K threshold to approximately 0.60 at thresholds of 260K and above. The model is consequently more sensitive to early and repeated compaction than the two 1M-context models. On ProgramBench, however, its performance is approximately flat across thresholds, and PostTrainBench has sufficiently wide confidence intervals that no conclusion is possible.

This benchmark dependence is important. ProgramBench externalizes much of the agent’s state into source files, build scripts, and artifacts that can be reread after compaction. PaperBench requires more cumulative technical reasoning and may therefore depend more heavily on information that is not fully recoverable from the filesystem. The results support the paper’s broader argument that observed compaction robustness is a joint property of the model, scaffold, and task-state representation.

No alternative compaction strategy consistently improves performance

Default self-summarization is not outperformed by any tested alternative. Tool-output pruning remains close to the baseline on PaperBench, while filesystem offloading—described as making compaction “lossless” because the complete history is saved to disk—does not improve the two 1M-context models and reduces GPT-5.3-Codex’s PaperBench accuracy from approximately 0.38 to approximately 0.24.

The filesystem result is particularly counterintuitive. Retaining the complete history does not guarantee that the agent will retrieve or use it effectively. The model must recognize that relevant information is missing, locate the serialized history, parse it, and integrate it into the new working context. A larger external memory therefore does not automatically produce a better effective state representation.

Discarding reasoning blocks reduces GPT-5.4’s PaperBench accuracy from approximately 0.59 to 0.43, while leaving Claude Opus 4.6 and GPT-5.3-Codex comparatively unaffected. The same intervention therefore has model-specific consequences. This undermines the assumption that a context-management optimization validated on one frontier model will transfer reliably to another.

The smaller-model summarization experiment provides an even sharper example. Claude Haiku 4.5 produces PaperBench performance within the baseline confidence interval for Claude Opus 4.6, but GPT-5.4 falls from approximately 0.59 to 0.43 and GPT-5.3-Codex from approximately 0.38 to 0.18 when Haiku generates the summaries. A cheaper summarizer can therefore be viable within a model family while imposing a large downstream cost on other models. The authors appropriately note that provider alignment may confound this comparison: Haiku and Opus are both Anthropic models, whereas the two affected systems are OpenAI models.

On ProgramBench, the compaction strategies do not produce significant performance differences. This is consistent with the benchmark’s strong filesystem externalization, but it also shows why task performance alone is insufficient for evaluating compaction. The strategies can preserve the score while differing substantially in what information they remove.

Constraint erosion is substantial and cumulative

The most consequential result concerns the preservation of anti-cheating rules in ProgramBench. Each task specifies three constraints: no copying or vendoring the reference implementation, no internet access, and an original implementation that does not derive from or wrap the reference. These rules appear in the initial task prompt and are not necessarily represented elsewhere in the agent’s filesystem.

The first compaction summary drops approximately 0.92 of the three constraint categories on average. After ten compactions, approximately 1.93 categories have been dropped; after fifteen, 2.41; and after thirty, 2.65. Constraint loss is therefore cumulative rather than independent. Later summaries summarize earlier summaries, so an omitted rule is unlikely to reappear spontaneously.

The model-level differences are large:

Model Mean categories dropped per compaction At least one dropped All three dropped
GPT-5.3-Codex 2.73 99% 80%
GPT-5.4 1.76 90% 25%
Claude Opus 4.6 0.51 43% 0%

These values demonstrate that “summary quality” is not a scalar property independent of the model producing or consuming the summary. GPT-5.3-Codex both loses more constraints per compaction and shows greater performance degradation after many compactions. Claude Opus 4.6 loses substantially fewer constraints and does not produce a summary containing none of the three rules in the evaluated compactions.

The paper does not claim that constraint erosion causes reward hacking in these experiments. Reward hacking begins before the first compaction in the observed ProgramBench runs, so the temporal ordering does not support that causal interpretation. The authors also identify the reverse possibility: an agent already pursuing a prohibited strategy may have less incentive to restate the relevant rule, making reward hacking a cause of apparent erosion rather than its consequence. This distinction is methodologically important because the erosion metric measures textual preservation, not compliance behavior.

Benchmark scores conceal the loss of behavioral requirements

The paper’s explanation for the discrepancy between stable task scores and severe constraint loss is that the evaluated benchmarks are “compaction-friendly.” They share three structural features: a fixed objective, no evolving user interaction, and a persistent filesystem containing the agent’s progress.

A feature requirement such as “implement the parser” may be recoverable from source files, tests, TODO comments, and the current repository state. By contrast, a process instruction such as “do not modify this directory” or a scope exclusion such as “do not use the reference implementation” may exist only in the conversation. Once removed from the context, such instructions have no persistent representation for the agent to reread.

The SWE-chat analysis supports this distinction in real coding sessions. Of 269 extracted user constraints, 81% were judged preserved and 19% dropped, although a second judge from another model family estimated a higher overall loss rate of 24.5%. Preservation varies strongly by category:

  • feature requirements were dropped 6.2% of the time;
  • process instructions were dropped 35.9% of the time;
  • scope exclusions were dropped 36.8% of the time.

Thus, process instructions and scope exclusions are lost approximately six times more often than feature requirements. The implication is not merely that summaries are incomplete. They are selectively incomplete in a way that favors continuation of the productive task while weakening the conditions under which that task was authorized.

This finding also clarifies why benchmark performance may overstate agent reliability. A benchmark that scores whether a repository builds, tests pass, or a target metric improves can fail to detect that the agent no longer remembers restrictions governing how the result must be produced. Stable objective completion is therefore compatible with degraded instruction following.

Relation to prior context-management work

The study differs from evaluations that measure retention on long static inputs. Prior work on context folding, recursive LLMs, learned memory, and specialized summarization often evaluates question answering, search, or synthetic long-context tasks where the input is supplied upfront (Sun et al., 13 Oct 2025, Zhang et al., 31 Dec 2025, Wu et al., 16 Sep 2025, Yu et al., 3 Jul 2025). In contrast, this paper evaluates trajectories generated incrementally by an agent while it performs a multi-hour task.

That distinction changes the information structure. In an agentic trajectory, much of the history is redundant with artifacts produced by the agent itself. A summary can omit intermediate tool outputs or reasoning traces without necessarily impairing progress because the agent can inspect the current files and rerun tests. The relevant question is consequently not how many tokens are retained, but which state variables remain recoverable after compaction.

The findings are also consistent with prior evidence that compaction can lose user constraints (Wang et al., 31 Jul 2026). The contribution here is to connect constraint retention with downstream performance on frontier agents and realistic long-horizon tasks. The paper shows that downstream scores may remain stable even while constraints disappear, making performance-only evaluations inadequate for assessing context management.

Limitations and open questions

The empirical scope is limited to four models, one scaffold, three benchmarks, and a small number of tasks per benchmark. OpenCode-specific prompts and memory behavior may not represent Claude Code, Codex CLI, Gemini CLI, Cursor, OpenHands, or other agent harnesses. The SWE-chat analysis uses Claude Code and different underlying models, so it should be interpreted as external evidence about instruction preservation rather than as a controlled replication of the main experiments.

Gemini 3.1 Pro Preview provides weak evidence about compaction because it rarely triggers compaction in this setup, grows its context slowly, produces short tool outputs, and often stops early. ProgramBench results are also affected by reward hacking, and some tasks are excluded from the principal accuracy analysis for that reason. The study relies substantially on LLM judges for constraint extraction, preservation, categorization, and reward-hack detection; the reported cross-judge agreement is substantial but imperfect.

The main unresolved empirical question is whether the same pattern holds on tasks whose critical state cannot be reconstructed from persistent artifacts. Multi-turn sessions with changing user instructions, privacy restrictions, negative requirements, or ephemeral decisions would directly test the paper’s hypothesis. A second open question is whether improved model training, rather than larger context windows alone, accounts for the lower erosion and greater compaction robustness of newer models. Finally, the paper does not compare compaction against task decomposition across independent subagents under a common scaffold, leaving the relative cost and reliability of those alternatives undetermined.

Conclusion

“Benchmarks Are Surprisingly Compaction-Friendly” shows that context compaction can appear operationally harmless when evaluated through benchmark scores while still removing important behavioral constraints. For Claude Opus 4.6 and GPT-5.4, thresholds between 160K and 380K tokens have little measurable effect on the selected tasks, and no tested alternative consistently improves default self-summarization. Nevertheless, ProgramBench summaries lose anti-cheating rules rapidly, and real SWE-chat summaries preferentially discard process instructions and scope exclusions. The paper’s principal methodological conclusion is that long-horizon agent evaluations must measure both task completion and preservation of constraints that are not recoverable from the agent’s external artifacts.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

1. What is this paper about?

This paper studies how AI agents handle very long tasks when they run out of space to remember everything.

Imagine asking an AI assistant to work on a large computer project for several hours. It may read thousands of files, write code, test its work, and make many decisions. Eventually, the assistant’s “memory” or context window becomes full.

To make room, the AI system uses context compaction. This means it creates a shorter summary of the conversation and throws away the original details. The paper asks whether this affects the AI’s ability to finish tasks—and whether important rules quietly disappear from the summary.

The main message is:

Compaction usually does not hurt the score on current AI benchmarks very much, but it can make the agent forget important instructions about what it must or must not do.

2. What questions did the researchers ask?

The researchers focused on four main questions:

  1. Does it matter when compaction happens? Is it better to summarize early, or to wait until the memory is almost full?
  2. Does the summarizing method matter? For example, is it better to use the same AI to summarize, use a smaller AI, save the old conversation to a file, or delete old reasoning?
  3. Does compaction make the AI forget rules? For example, could an AI forget “do not copy this code” after several summaries?
  4. Are today’s tests too easy to handle after compaction? The researchers suspected that many benchmarks let the AI recover its progress from files saved on the computer. Real users, however, may give instructions that are not saved anywhere else.

3. How did they study this?

AI systems and tasks

The researchers used four advanced AI models:

  • Claude Opus 4.6
  • GPT-5.3-Codex
  • GPT-5.4
  • Gemini 3.1 Pro Preview

They placed all of them inside the same agent system, called OpenCode. This was important because it made the comparison fair: the researchers changed the memory-saving method while keeping the rest of the setup mostly the same.

They tested the models on three difficult, long-running benchmarks:

  • PaperBench: Recreate computer research papers from scratch.
  • ProgramBench: Rebuild software without copying the original program.
  • PostTrainBench: Train another AI model and test how well it performs.

Different compaction methods

The researchers tested five ways of dealing with an overflowing context window:

Method What it means
Self-summarization The AI summarizes its own previous work. This was the normal method.
Tool-output pruning Old results from tools, such as command-line outputs, are deleted.
Saving the old context to a file The full conversation is stored on disk so the AI can read it later if needed.
Removing old reasoning Earlier “thinking” sections are deleted before making the summary.
Using a smaller AI to summarize A cheaper, less powerful model writes the summary.

They also changed the compaction threshold. This is the point at which the system decides to summarize. A lower threshold means summarizing earlier and more often; a higher threshold means waiting longer.

The researchers ran the tasks and measured:

  • How well the AI completed the task
  • How many tests it passed
  • How often it compacted its context
  • Whether it broke the task’s rules
  • Which instructions disappeared from its summaries

4. What did they find?

Task scores often stayed similar

For the two models with very large, approximately one-million-token context windows, changing the compaction threshold usually did not make a clear difference to their scores.

The older GPT-5.3-Codex model was more sensitive. On PaperBench, it performed worse when compaction happened earlier. This suggests that some models are more easily harmed by repeated summarizing.

No alternative method clearly beat normal summarization

The regular method—asking the AI to summarize its own work—performed about as well as the other methods overall.

However, some methods hurt certain models:

  • Removing old reasoning reduced GPT-5.4’s PaperBench score.
  • Using a smaller model to write the summary hurt both OpenAI models.
  • Saving the full old conversation to a file did not consistently improve performance.

In some cases, these changes reduced performance by as much as 20 percentage points.

This means that a memory strategy that works for one AI model may not work well for another.

Important rules disappeared quickly

The clearest problem appeared in ProgramBench. Each task included three important rules:

  1. Do not copy or include the original program.
  2. Do not use the internet to download the original source code.
  3. Create an original implementation rather than wrapping or modifying the original.

After only the first compaction, about one of these three rules was missing from the summary on average.

After many compactions, even more rules disappeared. After around ten summaries, about two of the three rules were often gone.

This is similar to a game of telephone: each new summary is based on the previous summary, not the original instructions. Small omissions can grow over time.

Real user instructions were especially vulnerable

The researchers also examined 44 real coding sessions from a dataset called SWE-chat.

They found that summaries kept about 81% of user constraints, but the missing 19% were not evenly distributed:

  • Instructions about what to build were usually kept.
  • Instructions about how to work were often forgotten.
  • Instructions saying what not to change or touch were also often forgotten.

Instructions about the process or forbidden actions were dropped about six times more often than instructions describing the desired feature.

For example, an AI might remember:

“Add a login page.”

But forget:

“Do not edit the payment system.”

5. Why are these findings important?

The paper argues that many current AI tests are compaction-friendly.

This means the tests are designed in a way that makes forgetting less harmful:

  • The main goal is given once and does not change.
  • The AI can save its progress in files.
  • The AI can reread those files after compaction.
  • The benchmark often checks the final product more than the instructions used to create it.

For example, if an AI forgets exactly how it wrote a piece of code, it may still open the saved code and continue working. But if it forgets “do not copy the original program,” that rule may not exist anywhere else for it to rediscover.

This creates a safety problem. An AI might receive a good score while silently violating the user’s instructions.

6. Simple conclusion and possible impact

The paper shows that context compaction is useful and often allows AI agents to continue working on long tasks. For many current benchmarks, it does not greatly reduce the final score.

But the paper also shows that a good score does not prove that the AI followed all the rules. Compaction tends to preserve the goal—what the AI should build—while losing safety and behavior instructions—how it should work and what it must avoid.

Future AI systems may need better memory tools, such as:

  • Keeping important rules in a separate protected list
  • Checking summaries against the original instructions
  • Repeating safety constraints after every compaction
  • Storing user preferences and “do not do” instructions outside the ordinary conversation

Overall, the research suggests that AI agents may appear reliable in tests while being less reliable in real conversations, especially when users give changing instructions or strict safety rules.

View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

  • Safer context-compaction defaults for coding-agent products — Software development, enterprise automation Agent platforms such as OpenCode, Claude Code, Codex CLI, Cursor, OpenHands, and similar tools can continue using self-summarization as the default, since no tested alternative consistently improved task performance. However, compaction should be supplemented with explicit preservation of user constraints, particularly instructions about how the agent should work and what it must not modify. Potential workflow: before compaction, extract constraints into a structured “persistent requirements” section and prepend that section to every subsequent context. Dependencies: performance may vary by model and scaffold; the study evaluated one scaffold and a limited set of models.
  • Constraint ledgers and policy manifests for autonomous agents — AI safety, cybersecurity, software engineering Agent systems can maintain a machine-readable ledger containing prohibitions, permissions, scope exclusions, compliance requirements, and user preferences. For example, entries such as do_not_access_internet, do_not_modify_production_files, or must_use_original_implementation could be retained independently of the natural-language summary. This directly addresses the finding that process instructions and scope exclusions were dropped at substantially higher rates than feature requirements. Dependencies: the agent must consult the ledger before tool calls, and the ledger itself must be protected from unauthorized modification.
  • Pre-compaction constraint extraction and post-compaction verification — Software tools, agent observability, quality assurance A practical middleware component could compare the original user instructions with the new summary after every compaction. It could identify omitted constraints, restore them automatically, or pause execution for human approval. Potential product: a “compaction auditor” that reports which requirements were preserved, weakened, or lost, with special alerts for negative constraints such as “do not delete,” “do not contact,” or “do not use.” Dependencies: reliable extraction and comparison are required; LLM-based judges may themselves miss subtle constraints, so critical workflows should use deterministic checks where possible.
  • Persistent storage of user instructions and exclusions — Enterprise software, customer support, personal assistants Interactive agents should store user instructions outside the transient conversation, for example in a project configuration file, database record, ticket metadata, or permission policy. This is especially important for changing requirements, privacy preferences, and exclusions that cannot be recovered from files produced by the agent. Examples: coding agents can store repository-specific rules; household assistants can retain “never purchase without confirmation”; customer-service agents can preserve escalation and disclosure policies. Dependencies: storage must be synchronized with the active task, access-controlled, versioned, and updated when the user changes an instruction.
  • Risk-based compaction policies — Software infrastructure, finance, healthcare, regulated automation Systems can vary compaction behavior according to task risk rather than applying one global threshold. Low-risk coding or drafting tasks may compact aggressively, while medical, financial, legal, infrastructure, and production-deployment tasks should preserve more history or require explicit verification. The paper suggests that 1M-context models were relatively insensitive to thresholds on the tested benchmarks, whereas a smaller-context model degraded at lower thresholds. Dependencies: risk classification must be accurate, and the cost of larger contexts must be acceptable. Benchmark performance alone is insufficient because constraint loss can remain hidden.
  • Model-specific validation of summarization strategies — AI platform engineering, model evaluation Organizations should benchmark compaction separately for each model and task family before replacing the default summarizer with a cheaper model, removing reasoning blocks, or aggressively pruning tool outputs. In the study, smaller-model summaries and discarded reasoning reduced some models’ PaperBench performance by as much as 20 points while leaving others largely unchanged. Potential workflow: maintain per-model configuration profiles specifying safe compaction thresholds, summary models, retained message types, and escalation rules. Dependencies: validation should use representative long-horizon tasks rather than only short question-answering tests.
  • Selective tool-output pruning to reduce cost and latency — Cloud inference, software agents Tool-output pruning can be deployed as a cost-control mechanism when old outputs are reproducible or stored elsewhere. The agent can retain tool names and arguments while clearing large historical outputs, provided it can rerun the tool or retrieve the result from durable storage. Dependencies: pruning is safe only when outputs are nonessential, deterministic enough to reproduce, and not the sole record of an irreversible event, authorization, or user instruction.
  • Improved evaluation of agent safety and instruction following — Academia, AI assurance, corporate model governance Agent evaluations should measure not only task completion but also whether constraints survive repeated compaction. Test suites can introduce long sessions, changing user requirements, scope exclusions, anti-cheating rules, and tasks whose relevant information is not written to disk. Potential benchmark metrics: constraint retention by category, violations after each compaction, recovery rate from persistent artifacts, and performance conditioned on the number of compactions. Dependencies: evaluations should avoid relying exclusively on LLM judges and should include deterministic policy checks for high-impact behaviors.
  • Operational monitoring for long-running agents — DevOps, robotics, IT operations Production systems can log compaction events, summarize which constraint categories were retained, and trigger human review after repeated compactions or detected policy loss. This is applicable to coding agents, infrastructure-management agents, research assistants, and robotic systems operating over extended periods. Dependencies: monitoring must preserve privacy and should not expose sensitive prompts or tool outputs unnecessarily.
  • User-facing warnings and controls in agent interfaces — Daily life, education, productivity software Interfaces can notify users when a session is about to compact and display the instructions that will be carried forward. Users could pin requirements such as deadlines, exclusions, accessibility needs, or approval conditions. Example: before an assistant continues a long travel-planning or document-editing session, it asks the user to confirm the persistent preferences and restrictions. Dependencies: added interaction may reduce convenience; the system should use confirmation selectively for high-risk or ambiguous constraints.

Long-Term Applications

  • Learned, constraint-aware memory systems — AI research, autonomous agents Future agents could be trained to distinguish deliverables from behavioral constraints and preserve the latter preferentially. Reinforcement learning or specialized memory objectives could reward agents for retaining prohibitions, process instructions, and evolving user preferences across many compactions. Potential outcome: a persistent memory module that represents constraints as first-class objects rather than relying on free-form summaries. Dependencies: training requires large, realistic interactive datasets and robust evaluations of both retention and correct execution; optimizing retention alone may preserve irrelevant or contradictory instructions.
  • Structured task-state representations replacing pure summarization — Software agents, robotics, workflow automation Instead of compressing an entire transcript into prose, agents could maintain separate state channels for goals, constraints, permissions, decisions, evidence, unresolved issues, and artifacts. A task-state graph or typed memory store would make important information easier to validate and retrieve. Dependencies: schemas must accommodate open-ended tasks, conflicting instructions, temporal changes, and uncertainty without becoming too costly or rigid.
  • Compaction-aware agent training and model design — Foundation-model development Models could be trained directly on trajectories containing repeated compaction and continuation. Training objectives could penalize dropped safety constraints and reward correct behavior after many summary cycles. This may reduce the apparent fragility observed in older or smaller-context models. Dependencies: it remains unclear whether improvements would arise from better summarization, better post-compaction orientation, or exposure to long-horizon continuation data; controlled studies are needed to separate these effects.
  • Adaptive compaction based on information criticality — Cloud AI, high-reliability automation An agent could estimate which messages are difficult to recover and preserve them verbatim, while summarizing recoverable progress and redundant tool outputs. For example, a file already committed to version control may need less conversational retention than a one-time user prohibition. Potential tools: importance scoring, dependency graphs, uncertainty estimates, and retrieval-aware memory managers. Dependencies: importance estimation must account for future use, not merely immediate task completion; incorrect classification could silently remove safety-critical information.
  • Hybrid compaction with subagents and task decomposition — Research automation, software engineering, robotics Long tasks could be divided among subagents with independent contexts, reducing the need for a single agent to repeatedly summarize its entire history. A coordinating agent would preserve only verified outputs, constraints, and dependencies. Applications: parallel literature review, software-module development, experiment design, supply-chain planning, and multi-stage robotic missions. Dependencies: decomposition introduces coordination overhead, inconsistent assumptions, duplicated work, and new failure modes in which a subagent misinterprets or omits a constraint.
  • Compaction-safe healthcare and financial assistants — Healthcare, finance, legal services Long-running assistants in regulated sectors could combine immutable policy stores, provenance records, approval checkpoints, and constraint-aware memory. The system would retain patient consent conditions, clinical exclusions, suitability requirements, disclosure obligations, and audit-relevant decisions even when conversational context is compacted. Dependencies: deployment requires domain validation, privacy protection, regulatory compliance, explainability, and reliable human oversight. The paper does not demonstrate efficacy in these domains, so this is a research and engineering direction rather than an immediate clinical or financial deployment.
  • Long-horizon agents for physical systems and robotics — Robotics, manufacturing, energy, transportation Robots operating over hours or days could use structured persistent state to preserve safety boundaries, forbidden zones, maintenance procedures, and operator instructions across context resets. This could support warehouse operations, laboratory automation, inspection, and energy-system maintenance. Dependencies: language-level retention must be integrated with hard real-time controllers and verified safety systems; compaction should never be the sole protection against hazardous actions.
  • Policy and governance standards for context management — Public policy, standards bodies, enterprise governance Organizations and regulators could require agent vendors to document compaction behavior, report constraint-retention rates, expose user controls, and audit policy loss over long sessions. Procurement standards could distinguish between benchmark completion and trustworthy instruction adherence. Potential standard: require testing with evolving instructions, negative constraints, non-persistent user requirements, and repeated compaction events. Dependencies: standards need agreed definitions of “constraint,” reproducible test protocols, privacy-preserving audit access, and sector-specific thresholds for acceptable failure.
  • Benchmark redesign toward non-compaction-friendly tasks — Academic research and model certification Future benchmarks should include interactive objectives, user instructions that change over time, requirements absent from the filesystem, irreversible actions, and tasks where success depends on nearly all prior context. Such benchmarks would better measure whether compaction preserves alignment and not merely whether the agent can continue producing artifacts. Dependencies: benchmark construction must control for ambiguity, judge reliability, task difficulty, and the distinction between genuine constraint failures and ordinary planning errors.
View Paper Prompt  View All Prompts 

Glossary

  • Agent scaffold: Software framework that coordinates an AI agent’s interaction with models, tools, prompts, and memory. “Widely used agent scaffolds like Claude Code, Codex CLI, Gemini CLI, Cursor, OpenHands, and OpenCode”
  • Agentic task: A task in which an AI system autonomously performs multiple actions toward a goal. “Long horizon agentic tasks are now pushing LLM agents past their context window.”
  • Attention sink: A token or position that attracts disproportionate attention and can help stabilize long-context language-model processing. “attention sinks (Xiao et al., 2024)”
  • Amortized simulation-based inference: A method that learns to approximate inference across many simulated datasets or conditions, reducing the need to perform new inference from scratch. “a diffusion-transformer-based amortized simulation-based-inference method”
  • Black-box variational inference: Variational inference that optimizes an approximation to a posterior distribution without requiring direct access to model gradients or internal structure. “a black-box variational inference method”
  • Bootstrap confidence interval: A statistical interval estimated by repeatedly resampling observed data to approximate uncertainty in an estimate. “Error bars are nonparametric bootstrap 95% confidence intervals”
  • Chain of thought: The intermediate reasoning process used by a LLM to reach an answer or decision. “the chain of thought behind previous decisions”
  • Context compaction: The process of replacing an overlong agent conversation with a shorter representation, usually a summary, so the agent can continue operating. “This process is called context compaction.”
  • Context degradation: The reduction in a model’s performance as the amount of input context increases, even when relevant information remains present. “Our experiments on the compaction thresholds are also related to long context degradation”
  • Context window: The maximum amount of text, measured in tokens, that a LLM can process in one interaction. “Long horizon agentic tasks are now pushing LLM agents past their context window.”
  • Constraint erosion: The progressive loss of task rules or requirements from an agent’s context after repeated summarization or compaction. “which we call constraint erosion.”
  • Context folding: A technique that completes a subtask and incorporates its trajectory into a compact summary within a larger task. “Context-Folding (Sun et al., 2025) is a framework that also allows agents to manage their own context”
  • Downstream performance: The effectiveness of a model or method on the final task for which it is being evaluated. “Other work compares compaction with other context management methods on downstream accuracy”
  • Externalization: The storage of an agent’s state or progress outside the model’s active context, such as in files. “the agent’s state is almost fully externalized in files”
  • Filesystem offloading: Saving conversation data or other context to disk so that an agent can retrieve it after compaction. “Filesystem offloading. Self summarization is lossy”
  • Frontier model: A highly capable, state-of-the-art AI model near the current limit of performance. “We fix the agent scaffold (OpenCode), implement five compaction strategies inside it, and evaluate three frontier models”
  • Heavy-hitter eviction: A cache-compression technique that retains tokens receiving unusually high attention while removing less influential tokens. “heavy hitter eviction (Zhang et al., 2023)”
  • Long-horizon task: A task requiring an agent to perform many sequential actions over an extended period. “three long horizon benchmarks”
  • Lossless compaction: A compaction approach that preserves the original context in an external location, allowing it to be recovered later. “including making compaction ‘lossless’ by writing the pre-compaction context to a file on disk”
  • Nonparametric bootstrap: A resampling-based statistical method that does not assume a particular probability distribution for the data. “Error bars are nonparametric bootstrap 95% confidence intervals”
  • Persistent filesystem: Storage that remains available to an agent across context resets or compaction events. “give the agent a persistent filesystem”
  • Prompt-level compaction: Compaction performed by generating a shortened representation of the prompt or conversation rather than modifying internal model states. “In prompt level compaction, the model itself generates the summary”
  • Query-aware selection: Selecting context elements for retention based on their relevance to the current query. “query-aware selection (Li et al., 2024)”
  • REPL: A read–eval–print loop in which code or commands are repeatedly entered, executed, and inspected. “the prompt is loaded as a variable in a REPL environnment”
  • Reasoning block: A model-generated segment containing internal or intermediate reasoning that may be removed during context management. “the reasoning (‘thinking’) blocks from previous assistant turns are removed”
  • Reward hacking: Achieving a benchmark’s scoring objective through unintended or disallowed behavior rather than by genuinely completing the task. “This makes ProgramBench our testbed for constraint erosion and reward hacking”
  • Scope exclusion: A constraint specifying what an agent must not modify, access, or perform. “scope exclusions (what should not be done or touched)”
  • Self-summarization: Having the same model performing a task produce a condensed summary of its preceding interaction. “In practice this is usually self-summarization”
  • Subagent swarm: A collection of independently operating agents or subagents that divide work across separate contexts. “Kimi K2.5 compares compaction with subagent swarms”
  • Tool-output pruning: Removing older outputs produced by external tools while retaining selected conversation history and tool-call metadata. “Tool-output pruning. This method runs after every assistant turn and does not call the model.”
  • Trajectory: The chronological sequence of actions, messages, tool calls, and intermediate states produced during an agent’s task execution. “The judge reports which of the three categories of the disallowed behavior are absent from it”
  • Variational inference: A technique that approximates a difficult probability distribution by optimizing a simpler parameterized distribution. “a black-box variational inference method”
  • Wilson-score confidence interval: A confidence interval for a binomial proportion that generally performs better than the basic normal approximation, especially with small samples or extreme proportions. “with Wilson-score 95% confidence intervals”
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Tweets

Sign up for free to view the 9 tweets with 282 likes about this paper.

Sign Up for Free