Prefix Sliding for efficient test-time scaling

Abstract: Test-time scaling uses extra test-time compute to improve performance, such as letting LLMs reason longer when solving a problem. As models keep the entire reasoning trace in memory via full attention, hard tasks that need long thinking can be prohibitively expensive. However, we find most intermediate reasoning tokens lose importance as the model continues reasoning. This calls into question whether retaining them is worth the cost. Based on this insight, we propose Prefix Sliding, which discards tokens during reasoning that are not part of the prefix or the window of the last few thousand tokens. The prefix has key instructions and tools available to the model, while the most recent tokens are the current reasoning the model is working on. This caps the total memory requirement regardless of how long the model reasons, allowing for efficient long-horizon test-time scaling. Without training, Prefix Sliding can make existing models 3x faster while maintaining performance. Training with Prefix Sliding using reinforcement learning can achieve better performance by enabling scaling to reasoning traces beyond a hundred thousand tokens. Ablations show Prefix Sliding outperforms summarizing intermediate tokens or vanilla sliding window. Our code is at https://github.com/Muennighoff/prefix-sliding

• The paper presents Prefix Sliding, an inference-time attention pattern that uses a fixed prefix and a sliding window to retain key tokens, which achieves constant decoding cost efficiency vs. unbounded costs.
• Standard sequential test-time scaling faces excessive computational time and memory burdens due to the retention of intermediate tokens in the KV cache; Prefix Sliding manages this issue by managing what is prioritized in memory.
• Comprehensive implementation of Prefix Sliding leads to roughly a threefold speed-up in transformers, which increases the production of longer reasoning trajectories with consistent benchmark performance.

Problem formulation and central contribution

“Prefix Sliding for efficient test-time scaling” (2608.26070) addresses a systems bottleneck in sequential test-time scaling. Reasoning models improve performance by generating extended chains of thought, but standard causal self-attention retains the complete generation history in the KV cache. Consequently, the cost of decoding each successive token increases with sequence length, while memory consumption grows without bound. This is particularly restrictive for reasoning trajectories extending to tens or hundreds of thousands of tokens.

The paper’s central claim is that full retention of intermediate reasoning tokens is unnecessary. The authors propose Prefix Sliding, an inference-time attention pattern that preserves two disjoint regions of the context:

1. The prefix, containing system instructions, the task prompt, tool specifications, and other globally relevant information.
2. A sliding window, containing the most recent reasoning tokens.

As generation proceeds, intermediate tokens outside the prefix and current window are evicted. If the prefix has length $P$ and the window has length $W$, the retained context is bounded by approximately $P+W$, independently of the total number of generated tokens. The method therefore changes the asymptotic decoding behavior from an unbounded per-token cost under full attention to a constant cost once the warm-up phase has completed.

The paper makes a consequential engineering claim: an existing full-attention LLM can use Prefix Sliding without additional training, achieve comparable reasoning accuracy, and run approximately three times faster. Training with the same attention pattern further permits reinforcement-learning rollouts exceeding 100,000 tokens.

Motivation from attention concentration

The method is motivated by an empirical analysis of attention distributions in Qwen3-1.7B on an AIME25 reasoning trace. Averaged over layers and heads, attention concentrates disproportionately on the initial prompt tokens and on the most recent tokens. The first few tokens function partly as attention sinks, while prompt tokens encode the task and available instructions. The terminal portion of the reasoning trace receives increasing attention because it represents the model’s current computational state.

Intermediate tokens receive substantially less attention as reasoning progresses. This observation is compatible with a computational interpretation of chain-of-thought: once a local subproblem has been resolved, the textual derivation of that subproblem may no longer be needed in its original form. Prefix Sliding exploits this temporal locality without attempting to identify individual important tokens dynamically.

Figure 1: Attention probabilities concentrate on the prefix and the most recent reasoning tokens, with lower importance assigned to much of the intermediate trace.

This concentration does not establish that intermediate tokens are universally redundant. It supports a task- and model-dependent approximation: the prefix supplies persistent global state, while the recent window supplies working memory. The distinction is important because a pure sliding window discards the prompt itself and therefore eventually loses information about the task, tools, and generation protocol.

Algorithmic design

During decoding, Prefix Sliding applies an attention mask whose permitted key positions are the union of the prefix and the current local window. The prefix remains fixed; the window advances monotonically as new tokens are generated. Unlike reset-based methods, Prefix Sliding does not repeatedly reprocess surviving tokens after context eviction.

The implementation uses a two-level filtering strategy compatible with FlashAttention. Intra-tile masking applies elementwise masks to tiles that partially intersect the permitted attention region. Inter-tile skipping avoids loading and computing tiles that lie entirely outside the prefix or sliding window. The resulting kernel approaches the throughput of ordinary sliding-window attention, with a modest additional cost associated with retaining the prefix.

The paper evaluates two treatments of positional embeddings. Continue PE preserves the original, monotonically increasing positions of generated tokens, allowing cached representations to be reused directly. Reset PE reassigns positions after the window advances, which would require recomputation and is substantially more complicated under teacher forcing. On AIME25, the authors report no meaningful performance difference between the two choices, and use Continue PE throughout.

Figure 2: Continue PE and Reset PE exhibit similar AIME25 performance for a sliding window of 2048 tokens.

The result makes Continue PE a practical default, but it also leaves open whether the equivalence persists across architectures, much longer windows, and tasks requiring precise positional reasoning.

Inference efficiency and test-time scaling

The principal systems advantage is the bounded decoding cost. Full attention incurs increasing computation and KV-cache traffic as the reasoning trace grows. Prefix Sliding performs attention only over a fixed-size prefix plus a fixed-size window; after the window is filled, each additional token has approximately the same attention cost.

Figure 3: Full attention becomes progressively more expensive with sequence length, whereas Prefix Sliding reaches a constant-cost decoding regime.

The experiments use Qwen3-1.7B, vLLM, FlashAttention, custom Hopper kernels, and a single 80 GB H100 for throughput measurements. The authors measure average wall-clock thinking time rather than FLOPs alone, correctly emphasizing that memory traffic and KV-cache capacity strongly affect user-visible latency.

Prefix Sliding exhibits a warm-up phase: until the generated sequence reaches the window size, its behavior is close to full attention. Thereafter, old reasoning tokens are evicted and throughput stabilizes. Full attention continues to slow as sequence length increases. With a window of 4096 tokens, the reported headline result is an approximately 3$\times$ speedup without retraining, while maintaining comparable benchmark performance.

The paper emphasizes that the speed advantage does not arise because Prefix Sliding produces intrinsically better tokens at equal sequence positions. Instead, it can generate more tokens within the same wall-clock budget. Thus, the comparison is fundamentally one of compute allocation: bounded attention permits longer reasoning trajectories under a fixed latency or memory constraint.

Figure 4: Prefix Sliding obtains a better time–accuracy trade-off because its bounded decoding cost allows more reasoning tokens to be generated within the same time budget.

This distinction matters for interpreting the result. Prefix Sliding is not shown to improve the conditional quality of every individual next-token prediction relative to full attention. Its benefit derives from preserving access to the task prefix and recent state while making additional sequential computation affordable.

Reinforcement learning with long rollouts

The paper extends Prefix Sliding to GRPO-based reinforcement learning. Long rollouts are often truncated because storing and backpropagating complete generations causes trainer out-of-memory failures. Prefix Sliding reduces the sampler-side memory requirement, but naive backpropagation through a 100,000-token trajectory would still be impractical.

The authors exploit the limited effective receptive field of stacked sliding-window layers. Although a theoretical receptive field can grow as $W \times L$, empirical information propagation is reported to be closer to approximately $1.5W$ because of layerwise bottlenecks. This motivates two training procedures:

• Chunked backpropagation, which processes the rollout in chunks and accumulates gradients.
• Truncated backpropagation, which computes the RL loss only on the final window while supplying several preceding windows as context.

In the reported implementation, a 100,000-token rollout with a 2048-token window can be reduced to an 8192-token trainer input. The first 6144 tokens provide context, while the loss is applied only to the final 2048 tokens. The authors use a four-times-window context multiplier because it substantially reduces the mismatch between generator and trainer log probabilities.

The KL divergence between generator and trainer log probabilities falls sharply when the trainer receives more context than the loss-bearing window. Passing only the final 2048 tokens produces a KL divergence above 0.1, while 4-times context is close to the result obtained by passing the full 16,384-token sequence. The residual mismatch is attributed partly to numerical differences between the custom generation kernel and the trainer’s FlexAttention implementation.

Figure 5: Truncated backpropagation with Prefix Sliding can attain performance comparable to full attention and full-sequence backpropagation.

The training experiments report that Prefix Sliding enables rollouts beyond 100,000 tokens and can achieve higher rewards under approximately equal memory budgets because it permits substantially longer trajectories. A 7B model trained with an 8192-token window and a four-times trainer-context multiplier performs comparably to a full-attention system when sequence length is controlled.

The implication is specific: the method is not merely an inference optimization. It can alter the feasible training distribution by preventing long, potentially useful trajectories from being discarded solely because they exceed a fixed rollout limit. However, the reported results do not establish that gradients from early portions of extremely long trajectories are unnecessary in general; they establish that the proposed truncated objective is sufficient for the tested setup.

Comparison with alternative bounded-cost methods

The authors compare Prefix Sliding with three methods that also avoid unbounded long-horizon decoding costs.

Last-$k$ eviction periodically deletes all but the most recent $k$ tokens. It can achieve high local throughput, but it reprocesses surviving tokens after each reset and risks discarding useful context. Small $k$ values can force the model to reconstruct lost reasoning; large values incur substantial recomputation.

Summarization compresses each context segment into a short model-generated summary. This can theoretically preserve information from the entire trace, but introduces an additional generation phase, summary-length and prompting hyperparameters, and duplicated processing when the summary is inserted into a new context. The paper’s ablation uses the model itself to produce summaries rather than an external summarizer.

Vanilla sliding-window attention has constant cost but eventually removes the original prompt. The resulting loss of task identity and tool instructions causes performance to flatten or deteriorate on long reasoning problems.

Prefix Sliding retains the persistent prefix while avoiding the recomputation and restart overhead of Last-$k$ and Summary. Under the paper’s AIME25 evaluation, with a maximum generation length of 262,144 tokens and a local window of 4096 tokens, Prefix Sliding provides the best performance–efficiency trade-off among these alternatives.

Figure 6: Prefix Sliding outperforms Last-$W$0, summarization, and vanilla sliding-window alternatives under long-horizon AIME25 evaluation.

This comparison supports the paper’s design principle that the relevant distinction is not simply “old versus new tokens.” The prefix and recent reasoning tokens have different functional roles, and preserving both is more effective than applying a uniform eviction rule.

Window size, positional handling, and evaluation protocol

The experiments sweep window sizes from 512 to 16,384 tokens. Larger windows generally provide greater information retention but reduce the asymptotic speed and memory advantage. The appropriate operating point therefore depends on the temporal structure of the task.

The benchmark suite includes GPQA, MATH500, and AIME25, with results averaged over 64 stochastic runs. Sampling uses temperature 0.6 and top-$W$1 0.95, and budget forcing controls the maximum thinking budget. Answers are verified with a lightweight verification library. These choices make the reported comparisons more statistically stable than single-sample evaluations, although they remain concentrated on mathematical and reasoning-oriented tasks.

The paper’s central empirical claim is strongest at long horizons. For short generations, Prefix Sliding provides little benefit because many sequences terminate before the window begins evicting tokens. The warm-up phase can consume nearly the entire generation on fast tasks. Accordingly, the method should not be interpreted as a universal low-latency replacement for full attention.

Limitations and open questions

The evidence is limited primarily to Qwen3 models, with most experiments using Qwen3-1.7B and reinforcement-learning results including a 7B model. The paper does not provide a broad cross-architecture study or compare against recurrent, state-space, hierarchical-memory, or other models that require pretraining or architectural adaptation. Its conclusions therefore concern compatibility with existing pretrained transformers rather than the general superiority of Prefix Sliding over all bounded-memory architectures.

Information loss remains the principal algorithmic limitation. LiveCodeBench requires a window of at least 16,384 tokens to match full attention in the training-free setting. The authors attribute this to coding trajectories in which a function implementation is followed by thousands of tokens of comments; when execution returns to the code, its beginning may have left the window. This is a direct counterexample to the assumption that intermediate tokens become irrelevant. It also indicates that the optimal window size depends on the structure of the task, not only on model size.

Agentic settings introduce further complications. Tool outputs, file contents, or web pages can flood the sliding window and evict relevant information. Multi-turn interactions raise an unresolved policy question concerning whether later user instructions should be appended to the persistent prefix or allowed to leave the window. The paper suggests guardrails, incremental reading, or learned adaptation, but does not evaluate these mechanisms.

Finally, the claim that training can teach models to adapt their commenting or information-management behavior remains unverified for the reported coding failure mode. The paper explicitly leaves open whether reinforcement learning can systematically reduce the required window size without impairing performance.

Conclusion

Prefix Sliding modifies causal attention by retaining the task prefix and a bounded recent-reasoning window. Its principal contribution is to make sequential test-time scaling asymptotically constant-cost while remaining directly applicable to pretrained transformers. In the reported experiments, it provides approximately 3$W$2 faster training-free inference, supports reinforcement-learning rollouts beyond 100,000 tokens, and outperforms several bounded-cost baselines. The method’s effectiveness depends on a task-dependent locality assumption that fails when important state is separated from the current reasoning by a long intermediate computation. Within that constraint, the paper presents Prefix Sliding as a simple systems intervention with substantial consequences for the feasible length of transformer reasoning trajectories.