Learning When Not to Attend Globally (2512.22562v1)

Abstract: When reading books, humans focus primarily on the current page, flipping back to recap prior context only when necessary. Similarly, we demonstrate that LLMs can learn to dynamically determine when to attend to global context. We propose All-or-Here Attention (AHA), which utilizes a binary router per attention head to dynamically toggle between full attention and local sliding window attention for each token. Our results indicate that with a window size of 256 tokens, up to 93\% of the original full attention operations can be replaced by sliding window attention without performance loss. Furthermore, by evaluating AHA across various window sizes, we identify a long-tail distribution in context dependency, where the necessity for full attention decays rapidly as the local window expands. By decoupling local processing from global access, AHA reveals that full attention is largely redundant, and that efficient inference requires only on-demand access to the global context.

• The paper introduces All-or-Here Attention, a binary routing mechanism that toggles between full causal and local sliding window attention, skipping over 93% of full attention operations.
• It employs a lightweight, learnable router using a Straight-Through Estimator and L1 regularization to optimize context selection per token and per head.
• Experiments demonstrate that selective global attention maintains near-baseline performance, highlighting significant potential for efficient inference in transformers.

All-or-Here Attention: Rethinking Context Selection in Transformer-Based LLMs

Introduction

This paper presents All-or-Here Attention (AHA), a token-wise, headwise binary routing mechanism for Transformers that allows each attention head to dynamically toggle between full causal attention and a fixed-size local sliding window attention. Unlike most prior approaches that approximate or compress full attention via soft gating, AHA takes a strictly conditional, discrete route. The motivation is that, analogous to human reading, most tokens require only local context while global context is essential predominantly for resolving long-range dependencies. The authors empirically demonstrate that over 93% of all standard full attention operations are redundant on typical language modeling tasks—replacing them with local attention yields no measurable loss on downstream task accuracy. The implications for both the understanding of model redundancy and efficient inference are significant, providing a minimalist lens for revising how Transformers operationalize context.

Architecture and Training Objective

AHA's architectural core is a lightweight, learnable router at each attention head. For each input token and head, the router projects the hidden state to a scalar, which is compared with a fixed threshold to yield a binary gate. Based on this gate, each head either executes full-sequence causal attention or a truncated, local sliding window attention over the recent $w$ tokens only.

Figure 1: The All-or-Here Attention architecture, where the router determines, per-head and per-token, whether to compute global or local attention.

The indicator function governing the gate is directly optimized via a Straight-Through Estimator (STE), making it possible to train the gating while keeping the hard binary decision during forward passes. Regularization is enforced through an L1 penalty on router scores, with a coefficient $\lambda$ balancing sparsity against language modeling performance. This design is plug-and-play: a fully pre-trained Transformer can be adapted to use AHA by fine-tuning, without expensive retraining from scratch.

Experimental Analysis

Benchmark Results

Experiments are conducted on the OLMo-2-0425-1B-SFT model across a spectrum of representative tasks: MMLU, HellaSwag, CSQA for classification; GSM8K, MBPP, and MultiNews for generative tasks. AHA is evaluated under several local window sizes ($w \in \{16, 32, 64, 128, 256\}$). The central findings are:

• With $w=256$, full task performance is retained ($102.5\%$ on average relative to vanilla full attention), while 93.3% of global attention operations are skipped.
• Even at $w=128$, AHA achieves $100.9\%$ of the baseline performance, with only $11.6\%$ of attention heads activating full attention.
• Restricting $w$ to smaller sizes ($w=16$ to $w=64$) yields a graceful performance degradation (bottoming out at $92.7\%$ of vanilla), indicating robust adaptability of the gating logic.
• Performance is not monotonic in the use of global attention: aggressive overuse (low regularization) does not yield benefit, providing strong evidence for the model’s ability to avoid redundant computation.

Critically, the gating mechanism dynamically compensates for context truncation: as the local window shrinks, the heads increase their use of global attention to maintain accuracy.

Context Dependency and Head Specialization

Visualization of per-layer, per-head full attention rates reveals highly sparse activation patterns.

Figure 2: Average full attention usage $\mu_f$ by layer and head, showing that only a minority of heads frequently trigger global attention.

Figure 3: Sorted headwise full attention usage, confirming that a small fraction of heads account for nearly all global context computation.

Key observations include:

• The distribution of global attention usage is sharply long-tailed—most heads almost never trigger global computation.
• A subset of heads are always-on (full attention per token); others operate globally with very low frequency (sometimes with intervals of thousands of tokens).
• This specialization provides compelling internal evidence for labor division, consistent with prior multi-head interpretability work.

The sparsity pattern suggests that the majority of local and syntactic dependencies are captured entirely by the local context and that full attention is reserved for resolving rare, genuinely long-range dependencies.

Comparative Perspective

Unlike hierarchical, sparse, or linear attention mechanisms, which typically require either complex indexing, soft gating, or multi-stage compression, AHA’s binary conditional routing is explicitly simple and directly interpretable. The method draws links to conditional computation and adaptive inference, but operates at a finer, mechanistically distinct granularity: per-head and per-token, rather than at the layer or module level.

AHA’s results also contrast with adaptive attention span methods, like Gated DeltaNet or Mixture-of-Attention-heads, by showing that soft or learnable span selection is not required to achieve meaningful performance improvements or compute savings; the critical bottleneck lies in discretely learning the rare necessity of global context retrieval.

Limitations and Implementation Challenges

The main limitation is practical: achieving real speedups on existing hardware is non-trivial due to headwise heterogeneity—most GPU kernels and attention accelerators are optimized for static, dense computation. Realizing AHA's theoretical savings in production requires specialized implementations capable of skipping, or batching, attention paths per head and token, as well as possibly adapting memory layouts for dynamic execution.

Moreover, this study is currently limited to OLMo-2; broader generalization to other architectures (e.g., Mamba, Longformer, or hybrids) and longer context scenarios is a logical direction. AHA could be extended to support multi-choice routing (e.g., to a set of window sizes, or incorporating memory modules), bridging to Mixture-of-Experts and adaptive depths.

Theoretical and Practical Implications

The primary theoretical implication is that Transformer self-attention is inherently over-provisioned: the standard, per-token full context computation is rarely required outside a minority subset of heads, and only for the construction of particular long-range dependencies. This greatly refines prior assumptions regarding the necessity of “global” access and supports a strong Occam’s razor argument for conditional computation and context management in scaling LLMs.

On the practical side, AHA directly enables a principled compute/performance trade-off, with explicit knobs for window size, regularization, and routing threshold. If hardware and frameworks adapt to support conditional attention execution, order-of-magnitude cost reductions for inference and training are plausible without quality degradation. This paradigm will be particularly salient as LLM contexts increase to tens or hundreds of thousands of tokens.

Conclusion

All-or-Here Attention provides an explicit mechanism for learning when not to attend globally, outperforming many traditional approximation schemes for efficient attention with a minimal binary gating approach. By empirically quantifying the redundancy in full attention computation and clarifying its sparse necessity, AHA reframes both theoretical and practical approaches to attention allocation in LLMs. Future work will likely generalize these findings to more granular or multi-choice attention routing and investigate opportunities for system-level optimizations to fulfill the substantial theoretical acceleration potential.