ProxyAttn: Guided Sparse Attention

• ProxyAttn is a training-free sparse attention method that refines block importance estimation by compressing along the head dimension instead of the sequence dimension.
• It employs dynamic, head-specific sparsity budgets to select critical token blocks, enabling efficient handling of long contexts such as 128K tokens.
• Empirical results show ProxyAttn outperforms existing sparse attention methods, achieving up to 10.3× speedup with minimal overhead.

ProxyAttn is a training-free sparse attention algorithm for long-context LLMs that targets the quadratic complexity of attention by improving block importance estimation in block sparse attention. Introduced in "ProxyAttn: Guided Sparse Attention via Representative Heads" (Wang et al., 29 Sep 2025), it departs from sequence-dimension compression and instead compresses along the head dimension, using pooled representative heads as proxy signals for block selection. The method is motivated by two empirical observations in multi-head attention: substantial similarity in token focus across heads, and heterogeneity primarily in head sparsity rather than in which tokens are prioritized. On that basis, ProxyAttn combines representative head pooling with block-aware dynamic budget estimation to obtain fine-grained block importance evaluation at low computational cost.

1. Motivation and problem setting

The central problem addressed by ProxyAttn is the quadratic complexity of the attention mechanism in Transformer-based LLMs, which becomes a major efficiency bottleneck for very long context windows such as 128K or more tokens (Wang et al., 29 Sep 2025). Existing sparse and block-sparse attention mechanisms accelerate inference by dynamically selecting important blocks, but the paper states that most rely on coarse-grained importance estimation strategies, including pooling over the sequence dimension. Under high sparsity, such strategies can miss critical but rare high-attention tokens inside large blocks, producing performance degradation.

The method is situated against existing techniques including Minference, FlexPrefill, XAttention, and SeerAttention. The paper characterizes these approaches as making heuristic assumptions, compressing along the sequence, or requiring additional training, and further states that they tend to suffer from accuracy loss at high sparsity. ProxyAttn is designed specifically to preserve the efficiency advantages of block sparse attention while improving the fidelity of importance estimation.

A useful way to frame the paper’s objective is as a refinement problem: block sparsity already reduces computation, but its effectiveness depends on how accurately one estimates which blocks are important. ProxyAttn addresses that estimation stage rather than redefining the attention operator itself. This suggests that its contribution is algorithmic and inference-oriented rather than architectural in the sense of requiring retraining or model modification.

2. Empirical basis: similarity across heads and heterogeneity in sparsity

ProxyAttn is built on two observations drawn from empirical analyses on Llama and Qwen models (Wang et al., 29 Sep 2025). First, most attention heads have consistent token focus, meaning that there is high overlap in the tokens attended to across different heads, especially in deeper layers. Second, the primary difference between heads is their level of sparsity: some heads attend narrowly, others more broadly, yet they tend to prioritize similar tokens.

These observations are important because they distinguish two different notions of diversity in multi-head attention. ProxyAttn does not assume that all heads are identical. Rather, it assumes that heads are sufficiently similar in token prioritization that a shared proxy can estimate block importance, while still allowing head-specific sparsity budgets. The paper explicitly states that even between sparse and dense heads, the important tokens are mostly shared; the main difference is in how many tokens are included.

The reported evidence includes Figure 1, which shows high overlap in the top tokens attended by different heads. The method’s design follows directly from that empirical pattern. If the salient token set is largely shared, then attention maps from pooled representative heads can serve as a low-cost estimator. If head sparsity nevertheless varies, then a single fixed budget across heads would be suboptimal. ProxyAttn therefore separates estimation sharing from budget allocation.

A common misunderstanding would be to treat head similarity as a claim that the multi-head structure is redundant in every respect. The paper does not make that claim. Instead, it argues that similarity is strong enough to support shared estimation, while head-specific sparsity remains necessary for accurate masking.

3. Representative head pooling and proxy-based block estimation

The first core component of ProxyAttn is attention score estimation via representative head pooling (Wang et al., 29 Sep 2025). Rather than compressing attention along the sequence dimension, ProxyAttn compresses along the head dimension. Heads are partitioned into groups $G_g$, and within each group a representative, or proxy, head computes attention scores that are then shared within the group for block importance estimation.

For group $g$, the pooled representative query and key states are defined as

$$\mathbf{Q}^g = \frac{1}{|G|} \sum_{i \in G} \mathbf{Q}^i\,, \quad \mathbf{K}^g = \frac{1}{|G|} \sum_{i \in G} \mathbf{K}^i$$

The group attention map is then given by

$$\mathbf{A}^i = \text{maxpool}\left( \text{softmax}\left( \frac{\mathbf{Q}^g {\mathbf{K}^g}^T}{\sqrt{d_k}} \right) \right), \qquad i \in G_g$$

where max-pooling is applied within blocks for fine-grained importance. This design preserves localized salience that sequence-wise compression may smooth away. In the paper’s framing, that is the key reason ProxyAttn can remain effective at higher sparsity levels.

To further increase efficiency, ProxyAttn applies stride-based downsampling: only the first token is retained per stride window when calculating $qk$ for importance estimation. If $n$ is the number of total heads and $g$ is the number of proxy heads, typically $1\!-\!4$, then the estimation cost is reduced to

$\frac{g}{n \cdot \text{stride}^2}$

that of full attention, which the paper describes as often about $1\%$ of the original.

This pooling mechanism is the methodological center of ProxyAttn. It uses shared proxy heads not to replace all downstream head-specific computation indiscriminately, but to guide sparse block selection. That distinction matters because the paper’s claim is about low-cost, fine-grained estimation rather than a full collapse of multi-head attention into a single shared map.

4. Block-aware dynamic budget estimation

The second core component is block-aware dynamic budget estimation, introduced to model the variation in sparsity across heads (Wang et al., 29 Sep 2025). Each head is assigned an independent budget, interpreted as the number of non-zero blocks. The paper estimates the required sparsity for head $g$0 online using queries from the last block.

The procedure is summarized as follows. For head $g$1, one computes the softmax of last-block query-key dot products, average-pools over blocks, and then selects the minimal number of top blocks $g$2 such that the cumulative probability exceeds a threshold $g$3. The resulting normalized ratio sets the target sparsity $g$4:

$g$5

This block-aware budget is then used to create head-specific Top-K masks:

$g$6

The stated consequence is that each head can have independent sparsity, preserving information in dense heads while maximizing speedup in sparse ones. In the logic of the method, representative proxy heads provide a shared estimate of where important content is likely to be found, while dynamic budgets determine how aggressively each head can prune.

The pairing of shared estimation with independent masking is the specific mechanism by which ProxyAttn reconciles the two empirical observations in the paper. Similarity among heads justifies shared proxy scores; heterogeneity in sparsity necessitates head-specific budgets.

5. Quantitative performance and efficiency

The paper reports that ProxyAttn consistently matches or outperforms existing block-sparse attention methods in both synthetic and real benchmarks for long-context LLMs, including RULER, InfiniteBench, and LongBench-v2, on Llama3.1-8B and Qwen2.5-7B (Wang et al., 29 Sep 2025). It is described as achieving the highest or nearly highest average and weighted-average accuracy, and sometimes even beating full attention. The paper also states that performance is retained at extremely high sparsity, upwards of $g$7 to $g$8 mask.

The quantitative efficiency claims are central:

Aspect	Reported result	Context
Attention acceleration	up to $g$9	256K token context length
Prefilling acceleration	up to $$\mathbf{Q}^g = \frac{1}{|G|} \sum_{i \in G} \mathbf{Q}^i\,, \quad \mathbf{K}^g = \frac{1}{|G|} \sum_{i \in G} \mathbf{K}^i$$0	end-to-end prefill / TTFT
Estimation overhead	< $$\mathbf{Q}^g = \frac{1}{|G|} \sum_{i \in G} \mathbf{Q}^i\,, \quad \mathbf{K}^g = \frac{1}{|G|} \sum_{i \in G} \mathbf{K}^i$$1 of full attention time	block importance estimation

The attention computation speedup is attributed to higher achievable sparsity rates. The end-to-end prefilling result is reported as a reduction in Time-To-First-Token under a realistic pipeline including MLP, not only kernel time. The paper further states that estimation overhead for block importance is negligible, citing an ablation in Figure 2.

Scalability is also emphasized. ProxyAttn is reported to remain effective for much larger models, including Llama3.1-70B with 64 attention heads, while still requiring only a single proxy head for accurate estimation. This is notable because the method’s rationale depends on cross-head similarity; the paper’s large-model result indicates that the same intuition remains useful even when the number of heads is substantially larger.

6. Relation to prior sparse-attention methods and broader implications

ProxyAttn is explicitly presented as a training-free alternative to existing methods such as Minference, XAttention, FlexPrefill, and SeerAttention (Wang et al., 29 Sep 2025). The paper states that, unlike Minference, XAttention, and FlexPrefill, it does not require model modification or search, and that it handles per-head sparsity with dynamic, block-aware budgets rather than using the same mask everywhere. It is also described as outperforming even trainable methods, specifically SeerAttention, at key sparsity-efficiency-accuracy tradeoffs.

In practical terms, the paper argues that ProxyAttn enables practical deployment of LLMs for contexts spanning hundreds of thousands of tokens. It further states that near lossless performance at substantial speedups becomes possible for existing, unmodified LLMs, reducing inference TCO and removing context length as a bottleneck. Because these statements are formulated as implications of the reported experiments rather than direct theoretical guarantees, they are best read as an empirical deployment claim tied to the tested models and benchmarks.

The paper also points to broader design implications. The observed similarity between heads suggests that future LLM designs may safely reduce computation in redundancy among heads, or structure groups of heads for efficiency. This is presented as a forward-looking implication rather than a proven architectural prescription. Likewise, the claim that ProxyAttn’s principle can be combined with new efficient attention kernels and adapted to decoding and generation phases indicates a research direction rather than a demonstrated result within the reported experiments.

From an encyclopedic perspective, ProxyAttn’s contribution can therefore be summarized as a specific answer to a recurring problem in long-context inference: coarse block selection is cheap but inaccurate at high sparsity; fine-grained selection is accurate but expensive. ProxyAttn addresses that tradeoff by using representative heads as proxy estimators and by allocating sparsity budgets per head, yielding a fine-grained block importance evaluation at low computational cost.