vAttention: Verified Sparse Attention (2510.05688v1)

Abstract: State-of-the-art sparse attention methods for reducing decoding latency fall into two main categories: approximate top-$k$ (and its extension, top-$p$) and recently introduced sampling-based estimation. However, these approaches are fundamentally limited in their ability to approximate full attention: they fail to provide consistent approximations across heads and query vectors and, most critically, lack guarantees on approximation quality, limiting their practical deployment. We observe that top-$k$ and random sampling are complementary: top-$k$ performs well when attention scores are dominated by a few tokens, whereas random sampling provides better estimates when attention scores are relatively uniform. Building on this insight and leveraging the statistical guarantees of sampling, we introduce vAttention, the first practical sparse attention mechanism with user-specified $(\epsilon, \delta)$ guarantees on approximation accuracy (thus, verified). These guarantees make vAttention a compelling step toward practical, reliable deployment of sparse attention at scale. By unifying top-k and sampling, vAttention outperforms both individually, delivering a superior quality-efficiency trade-off. Our experiments show that vAttention significantly improves the quality of sparse attention (e.g., $\sim$4.5 percentage points for Llama-3.1-8B-Inst and Deepseek-R1-Distill-Llama-8B on RULER-HARD), and effectively bridges the gap between full and sparse attention (e.g., across datasets, it matches full model quality with upto 20x sparsity). We also demonstrate that it can be deployed in reasoning scenarios to achieve fast decoding without compromising model quality (e.g., vAttention achieves full model quality on AIME2024 at 10x sparsity with up to 32K token generations). Code is open-sourced at https://github.com/xAlg-ai/sparse-attention-hub.

• The paper introduces vAttention, which unifies deterministic selection with adaptive random sampling to provide verified (ϵ, δ) error guarantees.
• It achieves superior quality–efficiency trade-offs by matching full-attention quality at up to 20x sparsity while significantly reducing inference latency.
• Empirical evaluations on Llama and Mistral benchmarks demonstrate robust long-sequence generation and near-linear speedups, particularly for CPU-hosted KV caches.

vAttention: Verified Sparse Attention for Reliable and Efficient Long-Context LLM Inference

Motivation and Problem Statement

The exponential growth in context length requirements for LLMs has exposed fundamental scalability bottlenecks in the Scaled Dot Product Attention (SDPA) operator, particularly during autoregressive decoding. As context length increases, the KV cache size grows, and repeated memory-bound reads—especially when the cache is offloaded to CPU RAM—lead to significant latency. Sparse attention, which restricts computation to a subset of tokens, is a promising mitigation strategy. However, existing sparse attention methods, primarily based on approximate top-$k$ or top-$p$ selection and sampling, lack consistent approximation guarantees and fail to expose a controllable quality–efficiency trade-off to practitioners.

Limitations of Prior Sparse Attention Methods

Deterministic top-$k$ selection assumes that a small set of tokens dominate the attention output, which is not universally valid—especially in deeper layers or for input tokens with distributed contextual dependencies. Sampling-based methods, such as MagicPig (LSH-based sampling), offer a statistical perspective but do not consistently outperform top-$k$ or provide explicit error guarantees. Empirical analysis (Figure 1) demonstrates that the optimal trade-off between quality and efficiency varies across heads and queries, and no single method is universally superior.

Figure 1: The number of tokens required for $p$-coverage and the local attention errors across token budgets, highlighting the non-uniformity of optimal sparse selection.

vAttention: Algorithmic Design and Theoretical Guarantees

vAttention introduces a hybrid sparse attention mechanism that unifies deterministic selection (sink, sliding window, top-$k$) with adaptive random sampling. The core innovation is the provision of user-specified $(\epsilon, \delta)$ guarantees on the relative error of the attention output, verified per head and per layer. The algorithm proceeds as follows:

1. Deterministic Selection: Identify heavy-hitter tokens via sink, local window, and top-$k$ selection (using any approximate top-$k$ method).
2. Residual Sampling: Uniformly sample from the remaining tokens, with the sample size adaptively computed to ensure the $(\epsilon, \delta)$ guarantee.
3. Attention Computation: Combine deterministic and sampled tokens, weighting sampled contributions by inverse selection probability, to approximate both numerator and denominator of SDPA.

The sample budget is derived using the Central Limit Theorem (CLT) or Hoeffding’s inequality, with empirical results showing that CLT-based budgets are significantly more efficient while maintaining robust error control (Figures 12, 13).

Figure 2: vAttention’s composition of deterministic and sampling-based selection, with adaptive budget computation and GPU/CPU cache retrieval.

Theoretical analysis establishes that if both numerator and denominator are approximated within $(\epsilon_1, \delta_1)$ and $(\epsilon_2, \delta_2)$, the overall attention output is guaranteed to be within $2(\epsilon_1 + \epsilon_2)$ relative error with probability $1 - (\delta_1 + \delta_2)$. In practice, vAttention employs a relaxation that focuses on denominator-only guarantees, justified by the compounding effect of bias in error propagation and the lower computational cost of denominator estimation.

Figure 3: vAttention exposes the $\epsilon$ parameter for user control, achieves SOTA quality–efficiency trade-offs, and demonstrates strong correlation between user-specified tolerance and observed error.

Empirical Evaluation

vAttention is evaluated on Llama-3.1-8B-Instruct, Deepseek-R1-Distill-Llama-8B, and Mistral-7B-Instruct-v0.3 across RULER, LongBench, Loogle, and AIME2024 benchmarks. Key findings include:

• Superior Quality–Efficiency Trade-off: vAttention combined with oracle top-$k$ or HashAttention consistently outperforms both, achieving up to 4.5 percentage point improvements in accuracy at 10% sparsity on RULER32K-HARD.
• Bridging Full and Sparse Attention: vAttention matches full attention quality with up to 20x sparsity across datasets, outperforming even oracle top-$p$ baselines (Figure 4).
• Long-Form Generation: On AIME2024, vAttention maintains full model quality for generations up to 32K tokens at 10x sparsity, demonstrating robustness in reasoning tasks.
• Efficiency Gains: When the KV cache is hosted on the CPU, vAttention yields near-linear speedup, as inference latency is dominated by KV cache reads (Figure 5).

  Figure 4: Pareto curves for quality and error vs. density, showing vAttention’s superior trade-off across datasets and baselines.

  

Figure 5: Near-linear inference speedup for Llama models with CPU-hosted KV cache using vAttention.

Implementation Considerations

vAttention’s index and budget computation are fully vectorizable and GPU-friendly. The method requires only partial KV cache access for budget estimation, which can be managed via a small random cache. When composed with approximate top-$k$ methods, auxiliary structures (e.g., bit caches for HashAttention) are lightweight and can reside on the GPU. For CPU-hosted KV caches, vAttention’s efficiency is further amplified, and dedicated CUDA implementations can yield additional speedups.

Ablation and Robustness

Empirical ablation (Figure 6) confirms that denominator-only approximation provides strong correlation between user-specified $\epsilon$ and observed error, validating the practical relevance of the relaxation. CLT-based budget computation is shown to be tight and efficient, while Hoeffding-based bounds, though more conservative, require significantly larger sample sizes.

Figure 6: Variation of density, relative error, and quality on RULER-32K tasks under denominator-only approximation.

Figure 7: CLT-based approximation analysis for denominator estimation, demonstrating efficient sample requirements.

Figure 8: Hoeffding-based approximation analysis, showing higher sample requirements for equivalent guarantees.

Implications and Future Directions

vAttention establishes a new paradigm for sparse attention in LLMs by providing explicit, verifiable control over approximation error. This enables reliable deployment of sparse attention in production systems, where quality–efficiency trade-offs must be transparent and tunable. The method’s adaptability per head and per query, combined with its theoretical guarantees, positions it as a robust solution for long-context inference and memory-constrained environments.

Potential future developments include:

• Extension to multi-modal and cross-attention settings.
• Integration with hardware-aware scheduling for further efficiency.
• Exploration of tighter concentration bounds and adaptive error allocation across layers.
• Application to training-time sparsity and model compression.

Conclusion

vAttention is the first practical sparse attention mechanism to offer user-specified, verified $(\epsilon, \delta)$ guarantees on approximation accuracy. By unifying deterministic and sampling-based selection, it achieves superior quality–efficiency trade-offs, robust long-sequence generation, and significant inference speedups. The method’s theoretical and empirical foundations make a compelling case for its adoption in scalable LLM deployment, and its design principles are broadly applicable to future advances in efficient attention mechanisms.