ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention

Abstract: Efficient attention algorithms are critical to mitigate the quadratic cost of attention in long-context workloads. Prior work utilises block-scaled quantisation techniques on Blackwell GPUs to move attention computation to 4-bit precision to accelerate inference. However, these techniques result in significant quality degradation in long-context settings. We show that the output impact of quantisation error is highly non-uniform and increases with the importance of each query-key interaction, concentrating functionally relevant error in a small number of attention blocks that contain the most important tokens. We propose ThriftAttention, a low-bit attention variant that delivers near-FP16 long-context quality at FP4 inference efficiency. This approach proceeds in two stages. First, a heuristic rapidly selects a small number of important query-key block pairs for FP16 precision. Second, the selected blocks are computed in FP16 and the remaining blocks in FP4, with both paths merged via online softmax into a single output. We demonstrate across long-context benchmarks and model families that by computing only 5% of query-key blocks in FP16, ThriftAttention recovers on average 89.1% of the FP4-to-FP16 performance gap. We show ThriftAttention's advantage grows with sequence length, mitigating the systematic FP4 quality degradation observed at longer contexts. The code is available at https://github.com/joesharratt1229/ThriftAttention.

• The paper introduces a selective mixed-precision FP4 attention mechanism that strategically upgrades only high-impact query-key pairs to FP16 to maintain output quality.
• Benchmarks reveal that promoting as little as 5% of block interactions reduces quality gaps by nearly 90%, achieving significant speedups compared to full FP16 computation.
• The approach leverages hardware-supported FP4 Tensor Cores to efficiently scale long-context inference while preserving near-FP16 performance.

ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention

Background and Motivation

The quadratic time and space complexity of standard attention mechanisms pose critical bottlenecks in long-context LLM inference, particularly as context lengths reach six-figure tokens. Recent advances have leveraged hardware-supported FP4 Tensor Cores (e.g., NVIDIA Blackwell) to drive throughput gains with 4x the arithmetic efficiency of FP16 instructions. The tradeoff is non-trivial: while block-scaled quantisation in FP4 can accelerate inference dramatically, it leads to pronounced and systematic quality degradation in output distributions, especially at extended sequence lengths. Sparsity methods that select only salient query-key pairs further reduce compute and memory but irrecoverably omit low-magnitude interactions, often resulting in unacceptable accuracy drops unless sparsity ratios are kept conservatively low.

ThriftAttention addresses this precision-efficiency conflict by quantifying the non-uniformity inherent in quantisation error across query-key blocks. The paper demonstrates that functionally critical errors concentrate in specific block pairs with the largest attention scores, motivating a selective mixed-precision scheme that strategically computes only a small fraction of high-impact interactions in FP16, while relegating the remainder to the fast FP4 path.

Figure 1: Architectural overview of the ThriftAttention mechanism, showing block-based precision routing and online softmax output fusion.

Algorithmic Formulation

ThriftAttention consists of a two-stage process:

• Block-importance scoring: For every query block $Q_i$ and key block $K_j$, a lightweight heuristic computes importance $\hat{S}_{ij} = \bar{q}_i \cdot \bar{k}_j$, where $\bar{q}_i$ and $\bar{k}_j$ are token-block means. The top-$k$ key blocks per query are assigned FP16 precision; all others are routed through FP4.
• Mixed-precision computation: Attention kernels are fused to process FP4 and FP16 blocks within each query. FP4 block matmuls use microscale quantization (NVFP4 E2M1 format + FP8 micro-scaling), consistent with SageAttention3. All blocks, regardless of precision, contribute to the output via an online softmax, merging both precision paths.

An explicit FP4 quantization error analysis reveals that dominant errors occur almost exclusively in block pairs with high attention mass, particularly at long contexts.

Figure 2: Heatmap of FP16-to-FP4 attention quantization error ($e = |P_{FP16} - P_{FP4}|$) across Qwen3-8B blocks, showing sharp error concentration.

Empirical Evaluation

Extensive benchmarks across popular model families (Llama3, Qwen3, Ministral 3) and standard long-context datasets (LongBench-v1, HELMET, RULER, PG-19) thoroughly evaluate accuracy and latency tradeoffs:

• Performance recovery: Promoting only $5\%$ of block pairs to FP16 reduces the FP4-to-FP16 NLL gap by $\sim89.1\%$; $10\%$ and $K_j$0 recover nearly all FP16 quality.
• Efficiency: ThriftAttention achieves up to $K_j$1 attention kernel speedup and nearly $K_j$2 end-to-end generation speedup compared to FP16, with latency nearly equal to full FP4.

  Figure 3: Pareto frontier plot showing ThriftAttention's approach to FP4 latency and near-FP16 output quality at 131k context length (Qwen3-8B).

• Context scaling: As sequence length increases, FP4 quality degrades sharply; ThriftAttention attenuates this effect, providing up to $K_j$3 relative $K_j$4NLL reduction at $K_j$5k tokens.

  Figure 4: Per-token NLL increase over FP16 baseline for Qwen3-8B at varying context lengths, illustrating ThriftAttention's mitigation of long-context degradation.

• Kernel speed: End-to-end speedups over FlashAttention-2 are substantial, both for prefill and decode phases.

  Figure 5: Kernel and end-to-end speedups over FlashAttention-2 for Prefill and Decode, highlighting efficiency gains.

Comparative Analysis and Core Claims

ThriftAttention's key distinction from pure sparsity approaches is its principled retention of all block interactions, albeit in low precision. Comparative tests at matched compute budgets show that ThriftAttention recovers vastly higher downstream scores compared to Quest and sparse top-$K_j$6 methods, where omitted blocks lead to catastrophic tail errors. The quantized FP4 pathway ensures even marginal contributions remain, enabling smooth quality degradation rather than hard evictions. The algorithm requires no retraining and is agnostic to model family.

Practical and Theoretical Implications

Selective mixed-precision attention exemplifies hardware-aware algorithm design, delivering inference acceleration without compromising model output distribution integrity. The approach is particularly pertinent for consumer-grade Blackwell GPUs, paving the way for high-throughput, long-context LLM serving. The method increases KV-cache memory footprint moderately; further refinement may address this overhead, especially for data-center deployments leveraging SM100 asynchronous features.

ThriftAttention's adaptability raises questions about optimal block-selection heuristics, potential integration with sub-byte attention training, and co-design with memory layout/serve architectures. The error concentration phenomenon may also inform future work in context-aware quantization, targeted pruning, and dynamic precision allocation.

Conclusion

ThriftAttention achieves selective mixed-precision attention for efficient FP4 inference at scale, preserving near-FP16 quality in long-context workloads with minimal overhead. Its training-free algorithmic pipeline and hardware-centric optimizations constitute a robust solution to the enduring quality-throughput dilemma in attention computation for LLMs.