Stochastic Additive No-mulT Attention (SANTA)

• SANTA is a stochastic sparse attention method that replaces dense value aggregation with unbiased post-softmax sampling to reduce memory bandwidth during decoding.
• It employs stratified and systematic sampling strategies to control estimator variance while maintaining performance on long-context autoregressive tasks.
• Empirical results demonstrate that SANTA can significantly cut value-stage operations—up to 90% reduction—while closely matching standard SDPA accuracy.

Searching arXiv for the primary SANTA paper and a small set of related methods mentioned in the provided material so the article can include accurate citations. Stochastic Additive No-mulT Attention (SANTA) is a stochastic sparse attention method for long-context autoregressive decoding that targets the memory-bandwidth bottleneck created by repeatedly streaming the key-value cache at each decode step. In standard scaled dot-product attention, a single query must read all $n_k$ key and value rows to compute

$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$

where $p_i$ denotes the post-softmax probability assigned to key $i$. SANTA replaces dense value aggregation with post-softmax sampling: it draws only $S \ll n_k$ indices from the categorical distribution defined by $p$ and aggregates the corresponding value rows using gather-and-add rather than dense multiply-accumulate. The method was introduced in “Stochastic Sparse Attention for Memory-Bound Inference” (Lee et al., 3 May 2026).

1. Decode-time motivation and computational setting

SANTA is motivated by the observation that autoregressive decoding becomes bandwidth-limited at long contexts because generating each new token requires reading the entire KV cache for all prior positions (Lee et al., 3 May 2026). Even with highly optimized exact kernels such as FlashDecoding and FlashInfer, bandwidth dominates once $n_k$ is large because the kernel must read essentially the full KV state at every step. The paper gives a concrete scale point: for Llama-3.1-8B-Instruct at 32k tokens, streaming bf16 $K/V$ is approximately 128 MB per layer per generated token, and KV traffic scales linearly in context length $n_k$ and batch size.

For a single head with query $q\in\mathbb{R}^{1\times d_k}$, keys $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$0, and values $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$1, dense attention incurs value-stage cost per head of $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$2 multiplications, $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$3 additions, and $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$4 memory reads for $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$5. Typical head dimensions are reported as $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$6–$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$7, while $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$8 may range from 8k to 32k or beyond. Grouped-query attention reduces footprint by sharing keys and values across queries, but it does not remove the requirement that the cache be read at every decode step.

The central significance of this setting is that dense exact attention kernels are already highly optimized arithmetically, so further latency reduction at long contexts depends increasingly on reducing memory traffic rather than merely improving fused multiply-add throughput. SANTA therefore targets the value stage as a sparse cache-access problem rather than as a conventional low-rank or deterministic truncation problem.

2. Core estimator and unbiasedness

SANTA treats the post-softmax weights as a categorical distribution over indices $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$9. Its default estimator uses i.i.d. draws with replacement:

$p_i$0

Because

$p_i$1

the estimator is unbiased for the exact post-softmax value aggregation (Lee et al., 3 May 2026). Sampling is implemented via inverse-CDF or categorical RNG, after which rows are gathered and added. The “No-mulT” label refers to the replacement of value-stage multiply-accumulate by gather-and-add, with only a final normalization by $p_i$2.

The paper distinguishes clearly between with-replacement and without-replacement designs. SANTA’s default implementation uses with-replacement sampling, together with variance-reduced stratified or systematic constructions that preserve the same expectation. If sampling without replacement is imposed, unbiased estimation of the same target can be obtained through a Horvitz–Thompson-style correction:

$p_i$3

where $p_i$4 is the sampled set and $p_i$5. The paper states that with-replacement sampling is simpler and GPU-friendly, and that the kernels use with-replacement or stratified/systematic variants with the same expectation.

This formulation implies a change in the asymptotic value-stage access pattern. Dense SDPA requires $p_i$6 value reads and the same order of additions and multiplications per head per query, whereas SANTA reduces this to $p_i$7 value reads, $p_i$8 additions, and only $p_i$9 multiplications or divisions for normalization. When $i$0 is much smaller than $i$1, the method reduces value-cache traffic by a factor of $i$2.

3. Variance, error behavior, and stratified variants

Let $i$3 and define

$i$4

For with-replacement sampling, the covariance of the estimator satisfies

$i$5

and the mean-squared error obeys

$i$6

The paper gives the bound

$i$7

which makes explicit that error decreases as $i$8 increases. It also states a qualitative dependence on the attention distribution: variance is smaller when attention is peaked and larger when $i$9 is flat or high-entropy (Lee et al., 3 May 2026).

A vector Bernstein tail bound is given under the condition $S \ll n_k$0, yielding probabilistic control of $S \ll n_k$1. The empirical interpretation in the paper is that variance scales approximately as $S \ll n_k$2 on log–log plots, which supports choosing $S \ll n_k$3 to satisfy task-specific error tolerance.

To reduce variance while remaining GPU-friendly, the paper introduces equal-mass stratification over the attention CDF and names the resulting family S$S \ll n_k$4ANTA. The CDF $S \ll n_k$5 over $S \ll n_k$6 is partitioned into $S \ll n_k$7 equal-probability strata $S \ll n_k$8. Two variants are defined:

$S \ll n_k$9

and

$p$0

Both are unbiased. For independent stratified sampling, the paper states

$p$1

with strict improvement unless all stratum means coincide. Systematic sampling is likewise unbiased and empirically shows similar variance, although no general dominance theorem is claimed.

The paper also presents a general stratified estimator with stratum masses $p$2 and $p$3 samples per stratum:

$p$4

Classical Neyman allocation is cited as suggesting $p$5, while the GPU kernels implement a proportional-allocation analogue across tiles based on tile mass. This suggests that SANTA is not only a sparsification mechanism but also a sampling-design problem in which estimator quality depends materially on how probability mass is partitioned.

4. Kernel architecture, bandwidth reduction, and implementation strategy

A central challenge for stochastic attention on GPUs is that sampling indices from the post-softmax distribution typically requires global softmax mass information to build a CDF. The paper therefore proposes two CUDA kernel strategies for parallelizing stochastic attention (Lee et al., 3 May 2026).

The first, S$p$6ANTA-prop, is a two-pass design with a global synchronization. In pass 1, the sequence dimension is split into tiles of length $p$7, and per-tile exponentiated scores or equivalent local partition functions $p$8 are computed and written to global memory. The paper emphasizes that stashing scores is cheap because this stores only $p$9 scalars rather than $n_k$0 value vectors. A global budget allocation step then sums the tile masses to obtain the global partition function and assigns sample counts per tile in proportion to tile mass. Tiles with negligible mass receive $n_k$1 and skip value reads. In pass 2, systematic sampling is performed within each tile using the stashed scores and allocated budget, integer counts $n_k$2 are computed through stratified or systematic thresholds, and accumulation proceeds as $n_k$3, followed by normalization by $n_k$4. The reported benefit is exact load balancing across tiles with maximal pruning of low-mass tiles; the reported cost is one global synchronization.

The second, S$n_k$5ANTA-flash, adopts speculative uniform per-tile sampling with deferred normalization. Each tile samples as if the whole mass were local, using a roughly uniform budget $n_k$6. A merge kernel then computes the true global normalization and rescales tile partials:

$n_k$7

with

$n_k$8

This removes the global barrier and follows a FlashDecoding-style split–reduce pattern, but the paper identifies “sample waste” on low-mass tiles as the key drawback, since comparable accuracy then requires larger $n_k$9.

The implementation discussion attributes speedups primarily to reduced value reads. At long context, value-cache streaming dominates decode time, and S$K/V$0ANTA-prop with sparse reads such as $K/V$1 at 32k tokens yields greater than 90% reduction in value-stage traffic per query in single-head terms, together with similar arithmetic savings because the aggregation is mostly additive. Throughput-oriented details listed in the paper include contiguous gathers within tiles, shared-memory accumulation of counts $K/V$2, on-the-fly RNG for systematic offsets, explicit handling of zero-budget tiles to avoid warp divergence, stashing scalar score statistics instead of vector values, and a largest-remainder integer allocator for tile budgets.

The practical implication is that SANTA’s gain depends not only on estimator sparsity but also on whether that sparsity can be mapped onto tile-local, coalesced, low-divergence GPU execution. The paper’s kernel design treats sampling as an integral part of attention implementation rather than as an abstract estimator layered on top of an unchanged dense kernel.

5. Bernoulli $K/V$3 sampling and system-level integration

The paper presents Bernoulli $K/V$4 sampling as a complementary score-stage sparsification technique. Its mechanism is to represent the normalized query as sparse ternary random variables in order to prune low-importance features and reduce key-feature reads during decoding (Lee et al., 3 May 2026). After normalizing $K/V$5 by a scalar norm at least as large as $K/V$6, each element satisfies $K/V$7. For samples $K/V$8, one draws independent ternary Bernoulli vectors:

$K/V$9

The score estimator is then

$n_k$0

which is unbiased for $n_k$1. Because $n_k$2 is ternary, the resulting matrix product becomes multiplier-free in the sense that it can be implemented by gathering selected rows of $n_k$3, adding or subtracting them, and then applying normalization.

For grouped-query attention, the paper notes that the union of accessed features can approach dense if each query is sampled independently. It therefore proposes a mean group query:

$n_k$4

sampled via Bernoulli on $n_k$5, and then estimates each head’s score as

$n_k$6

This prunes features common to the group while compensating through elementwise scaling by $n_k$7.

The empirical examples reported for the score stage are model-dependent. For $n_k$8 in BitNet-2B, group key-feature access drops to approximately 84.7% with small accuracy loss. For Llama 8B at 4k, $n_k$9 reduces key-feature access to approximately 72.8% with less than 2% accuracy degradation on NIAH and QA tasks. The paper further states that stratified Bernoulli sampling reduces variance and that the $q\in\mathbb{R}^{1\times d_k}$0 error of $q\in\mathbb{R}^{1\times d_k}$1 scales approximately as $q\in\mathbb{R}^{1\times d_k}$2 in aggregate, consistent with Monte Carlo intuition.

SANTA and Bernoulli $q\in\mathbb{R}^{1\times d_k}$3 are described as orthogonal to upstream methods such as ternary or weight quantization, low-rank projections, KV-cache compression or quantization, grouped-query attention, and cache-management policies (Lee et al., 3 May 2026). In this framing, SANTA reduces the number of accessed value elements in whatever cache representation is retained, while Bernoulli $q\in\mathbb{R}^{1\times d_k}$4 reduces the number of accessed key features. The paper therefore positions the combination as a route toward sparse, multiplier-free, and energy-efficient inference rather than as a replacement for compression or architectural changes.

6. Empirical behavior, limitations, and relation to other sparse attention approaches

On an NVIDIA RTX 6000 Ada, using single-step decode attention kernels with Llama-3.1-8B tensor shapes at 32,768 tokens, the paper reports that S$q\in\mathbb{R}^{1\times d_k}$5ANTA-prop with $q\in\mathbb{R}^{1\times d_k}$6 achieves approximately $q\in\mathbb{R}^{1\times d_k}$7 speedup versus FlashInfer and FlashDecoding while matching SDPA accuracy on RULER long-context tasks within confidence intervals (Lee et al., 3 May 2026). Under the same setting, S$q\in\mathbb{R}^{1\times d_k}$8ANTA-flash achieves approximately $q\in\mathbb{R}^{1\times d_k}$9 speedup but requires a much larger budget, $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$00, to match accuracy because of speculative sample waste.

The 32k RULER results are reported as follows. For S$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$01ANTA-prop with $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$02, the scores are FWE $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$03, NIAH $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$04, QA1 $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$05, and QA2 $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$06. For S$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$07ANTA-flash with $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$08, the scores are FWE $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$09, NIAH $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$10, QA1 $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$11, and QA2 $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$12. The SDPA baseline is FWE $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$13, NIAH $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$14, QA1 $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$15, and QA2 $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$16.

At 8k-token contexts, S$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$17ANTA-strat and S$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$18ANTA-sys with $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$19 recover SDPA accuracy within approximately 1% across FWE, NIAH, and QA tasks while using only $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$20 of value-stage reads and additions in the single-query, single-head view. For general reasoning benchmarks in a PyTorch reference implementation, the paper states that on GSM8K with Llama-3.1-8B, S$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$21ANTA-strat and S$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$22ANTA-sys approach SDPA accuracy at $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$23, and that S$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$24ANTA-sys at $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$25 reaches 76.42% versus SDPA 78.06% while using approximately 19% of value-stage operations and reads and no multiplications. On MMLU with the same model, S$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$26ANTA-strat and S$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$27ANTA-sys match or exceed SDPA near $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$28 and outperform default SANTA across $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$29–$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$30.

The paper also identifies several failure modes and sensitivity patterns. Variance is higher when attention is flat, and stratified or systematic sampling improves robustness at small $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$31. Layer-sensitivity ablations using one-hot attention with $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$32 at a single layer show that certain layers are fragile, including Llama-8B’s first layer and middle layers in DeepSeek-7B, which motivates per-layer scheduling of $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$33. The prefill regime is explicitly described as unfavorable for sparse value access because the union over many queries covers most value rows; the paper gives an example in which more than 94% of unique rows are read at $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$34 across 1k–16k contexts with Gaussian inputs. This suggests that SANTA’s memory-sparsity benefits are fundamentally decode-centric even though it still reduces additions and eliminates multiplications during prefill.

The comparison to other sparse attention families is framed carefully. Deterministic top-$$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$35 attention reduces reads but is described as biased and potentially brittle on heavy-tailed or flat attention distributions, whereas SANTA is unbiased with controllable variance and shows stronger robustness at small budgets. Reformer, BigBird, Longformer, and Linformer are characterized as methods that restructure attention patterns or reduce complexity mainly in prefill, whereas SANTA targets decode-time bandwidth with post-softmax, distribution-aware sampling and preserves the exact distribution’s expectation (Lee et al., 3 May 2026). The paper further states that importance-sampling approaches often operate pre-softmax, while SANTA’s novelty lies in post-softmax sampling that converts the value stage into multiplier-free aggregation of sampled value rows.

The limitations listed in the paper include sample waste in speculative kernels, higher variance in flat-attention regimes, the limited latency advantage of multiplier-free arithmetic on current GPUs optimized for dense FMAs, and the absence of strong prefill memory savings. Future directions named explicitly are tighter mass-aware scheduling without global barriers, hardware RNG and systematic samplers, adaptive per-layer $$y=\mathrm{softmax}\!\left(\frac{qK^\mathsf{T}}{\sqrt{d_k}}\right)V =\sum_{i=1}^{n_k} p_i V_i,$$36 selection via reinforcement learning or heuristics, end-to-end kernels combining Bernoulli score-stage sparsification with value-stage SANTA, and integration with KV compression for compound savings. Within that scope, SANTA is presented as a decode-oriented stochastic attention framework whose defining properties are unbiased post-softmax estimation, sparse value-cache access, and an implementation strategy aligned with bandwidth-dominated long-context inference.