You Don't Need Long Context
This presentation examines whether long-context language models are truly necessary for common workloads that generate large prompts. The research demonstrates that frequent context compaction and selective input access can preserve task performance while substantially reducing peak context length, inference cost, and latency. Through systematic evaluation of long-horizon agentic tasks and long-input information processing, the work shows that many tasks requiring hundreds of thousands of input tokens can be solved with active contexts of 16K tokens or less, yielding deployment cost reductions of up to 8.8x while maintaining competitive accuracy.Script
Deploying a language model with a 256K token context window can reduce decode throughput by 7 times compared to a 16K window. That cost gap is larger than most quantization or sparsity techniques can achieve, yet the assumption persists that long prompts require long contexts.
The researchers propose treating context as actively managed computational state rather than a passive transcript. A software engineering task may generate thousands of commands and tool outputs, but only a small state representation is needed for the next action.
Three context regimes were tested. Full history accumulation keeps every action and observation in one conversation. End of conversation compaction produces a bounded state recap once a threshold is reached. Single action compaction discards the conversation after each environment action, transferring only semantic state forward.
Across long horizon benchmarks, accuracy remained within a few percentage points of baseline while estimated deployment cost fell by up to 8.8 times. Claude Opus improved from 70% to 75% on SWE bench Verified under an 18K compaction threshold, but Qwen Coder Next collapsed from 53% to under 4%, showing that robustness varies across model families.
For long input tasks, placing complete source material in the agent's filesystem rather than the initial prompt converted the problem into selective reading. On the LOONG benchmark, selective reading at a 16K limit improved Claude's accuracy from 70% to 79% and perfect answer rate from 39% to 53%, while reducing modeled cost by more than 200 times.
Single action compaction preserved SWE bench Verified accuracy almost exactly for Claude, with median peak context of only 9K tokens. This result suggests that models can maintain useful sequential behavior with compact semantic state, though whether dedicated state space training outperforms full history training remains an open question. Explore more research like this and create your own videos at EmergentMind.com.
