---
title: 8-Bit Quantized Attention
url: https://www.emergentmind.com/topics/8-bit-quantized-attention
type: topic
---

# 8-Bit Quantized Attention

8-bit quantized attention refers to the implementation of attention mechanisms in deep learning models where the involved tensor operations and intermediate representations are confined to 8-bit numerical formats, either in fixed-point or floating-point arithmetic. The primary motivation is reduced memory footprint, increased computational throughput, and lower energy consumption, particularly significant for large-scale transformer models and memory-augmented neural networks where attention is a computational bottleneck. Several algorithms and hardware-software co-designs have been developed to ensure that 8-bit quantization remains accurate and robust, even as the quantization may introduce nontrivial numeric error if not managed carefully.

## 1. Core Quantization Schemes and Formats

Two principal schemes dominate 8-bit quantized attention: fixed-point (Q-format) and floating-point formats optimized for hardware.

- **8-bit Fixed-Point (Q-format):** In memory-augmented networks, vectors (inputs, weights, and memory contents) are encoded as 8-bit fixed-point values with a signed magnitude, configurable for integer and fractional precision. The quantizer maps a continuous value $x$ in $[x_\text{min}, x_\text{max}]$ to
  \[
    Q(x) = \mathrm{round}(\mathrm{clamp}(x,x_\text{min},x_\text{max}) \cdot 2^{\mathrm{FRAC}}) / 2^{\mathrm{FRAC}}
  \]
  Proper range calibration (choice of integer/fractional split) is essential to prevent overflow while maintaining adequate precision. Binary quantization (sign-based activation) can be used for activations, with memory and parameters remaining in 8-bit fixed.

- **Block-wise and Vector-wise Quantization:** In transformer attention and projection layers, as seen in LLM.int8() and SageAttention, per-row (vector-wise) or per-block scales map tensor values linearly to signed 8-bit integers. For two tensors $X$ (inputs) and $W$ (weights), this results in quantized representations:
  \[
    X_\text{int8}[i,k] = \left\lfloor c_{x,i} \cdot X[i,k] \right\rceil, \quad c_{x,i} = 127/\|X[i,:]\|_\infty
  \]
  followed by dequantization after integer arithmetic.

- **8-bit Floating-Point (HiF8):** To address the needs of softmax computation (with its large dynamic range), BAPS proposes the HiF8 format (E5M2 or E4M3 selectable): 1 sign, 5 exponent, and 2 mantissa bits, supporting range $\approx 2^{-16}$ to $2^{16}$, sufficient for post-shifted softmax logits and exponentials. This enables all softmax computations, including exponentiation, in 8 bits without catastrophic underflow.

## 2. Algorithmic Adjustments to Attention Pipeline

Conventional attention computes $S = Q K^\top / \sqrt{d}$, $P = \mathrm{softmax}(S)$, and $O = P V$. In 8-bit pipelines, all major steps are modified to accommodate quantization:

- **Similarity Computation:**
  - Linear attention projections use vector/block-wise quantized matrix multiplies, with special handling for systematic "outlier" attention features (e.g., in LLM.int8(), which routes outlier dimensions to higher-precision arithmetic) [2208.07339].
  - In memory-augmented models, Q-MANN replaces dot/cosine similarities (prone to overflow) with bounded Hamming-based similarities computed by weighted XNOR and popcount on the bit representations, avoiding out-of-range accumulations [1711.03712].

- **Softmax Normalization:**
  - In BAPS and SageAttention, per-block rescaling is applied by dividing scores in each block by their maximum, so quantization is lossless in the range $[-1,1]$ before exponentiation.
  - Low-precision exponentiation (typically via LUTs or ROMs) computes all exp/softmax steps in 8 bits; cumulative sums are done at higher precision (16 or 32 bit), but the outputs are returned to 8 bits.
  - For robustness, block-wise rescaling and rare "restarts" to higher precision are implemented (restart ARR rates of 5–10% are empirically sufficient) [2602.02071].

- **Accumulation and Output:** SageAttention optionally leaves the $PV$ matmul and accumulator in FP16 for maximal fidelity, but as hardware improves, fully INT8 end-to-end is possible. Adaptive per-layer selection allows dynamically switching between INT8 and mixed kernels to maintain quality.

## 3. Quantization Error Analysis and Mitigation

The quantization process introduces two primary sources of error:

- **Similarity Measure Expansion:** Quantization error in inner products or distances is proportional to the quantization bin size and increases with dynamic range. For fixed-point Q-MANN, quantization error propagates exponentially through softmax, so the error in $s_t(i)$ causes error in attention weights $a_t(i)$ bounded by $\exp(\epsilon_{\text{max}})$.

- **Error Control Techniques:**
  - **Bounded Similarity:** Replacing unbounded similarities with bounded Hamming similarity ensures that quantized similarity values never overflow, even in pathological cases [1711.03712].
  - **Outlier Routing:** LLM.int8() identifies outlier dimensions that break the assumptions of uniform quantization and routes those separately to higher-precision matmuls, ensuring the remaining >99.9% of features are quantized aggressively with minimal global error [2208.07339].
  - **Block-Rescaling:** BAPS rescales each block before quantization to match the 8-bit range, and later undoes the scaling after computation. This keeps quantization errors bounded and prevents most catastrophic information loss [2602.02071].
  - **Gradient Smoothing and Mixed-Precision Backward:** During training, 8-bit quantization is extended to backward passes (SageAttention3), keeping only the $dO V^\top$ gradient mat-mul in FP16 and quantizing the remaining steps to INT8, which maintains final gradient similarity >99.7% and prevents error accumulation in longer sequences [2505.11594].

## 4. Hardware Optimization and Implementation Details

8-bit attention is tightly coupled to hardware-software co-design to exploit tensor-core accelerators and minimize bandwidth constraints:

- **Memory and Bandwidth:** INT8 storage for $Q$, $K$, $V$, and $P$ halves memory bandwidth compared to FP16, directly doubling throughput in most architectures. The largest memory and bandwidth bottlenecks, notably the transfer of $QK^\top$, are now fully quantized.

- **Exponentiation Units:** HiF8-based softmax exponentiation engines use table-based or simple fixed-function logic, reducing area by 3–5× compared to FP16 and further reducing cycle latency (1–2 vs. 4–6 cycles per op for softmax) [2602.02071].

- **Kernel Fusion:** In SageAttention, quantization is fused into prior layers (e.g., rotary embedding, previous linear transform), so overhead is minimal. Kernels dynamically switch between per-block (B/T) and fully quantized (vT/vB) operation depending on measured cosine similarity to FP16 [2410.02367].

- **Block-Tiling and Scheduling:** All major algorithms tile sequences into blocks of length 128–256 (FlashAttention-style), with per-block quantization and scaling. Micro-pipelining and ping-pong scheduling are used to maximize occupancy and overlap data movement with computation [2410.02367, 2505.11594].

## 5. Empirical Results and Evaluation

End-to-end empirical evaluations across multiple models and tasks show:

| Approach         | Speedup vs FP16 | Accuracy Δ (NLP)   | Memory Savings | Notes                                          |
|------------------|-----------------|--------------------|----------------|------------------------------------------------|
| Q-MANN           | ~20× energy     | +5–10 pp vs. naive | Up to 22×      | Hamming sim., robust to quantization [1711.03712] |
| LLM.int8()       | 1.6–2.3× (matmul) | <0.2% abs. (PPL/ZS) | 2× (weights)   | Outlier routing for zero-loss [2208.07339]      |
| SageAttention    | 2.1–2.7× (TOPS) | <0.5% any metric   | 2× ($Q$, $K$)  | Cosine sim. >0.9999, plug-and-play [2410.02367] |
| BAPS (HiF8)      | ~2× throughput  | ≤1 pt on NLP, ~0 on MM | N/A        | Softmax unit area 3–5× smaller [2602.02071]    |
| SageBwd (train)  | 1.67× overall   | ≤0.5% on FT tasks  | N/A            | Lossless fine-tuning, slower PT convergence [2505.11594] |

On scale, LLM.int8() enables deployment of 175B-parameter models on commodity GPUs with no empirical loss in perplexity or attention quality, and SageAttention delivers real speedups (2x–5.9x) across language, image, and video tasks while matching full-precision validation metrics.

## 6. Limitations, Trade-offs, and Best Practices

- **Precision vs. Overflow:** In fixed-point quantization, improper calibration of integer/fractional bits leads to either overflowing (catastrophic zeros) or excessive quantization noise (coarse bins). Careful per-block or per-vector scaling, and dynamic outlier handling, are required.
- **Approximation Effects:** Bounded similarity and reduced-precision math can degrade precision in very fine-grained attention tasks; weight constants and block sizes require per-task tuning.
- **Training vs. Inference:** 8-bit quantized attention for inference is essentially lossless with proper mitigation, but training exhibits slower convergence in large-scale pretraining, even though fine-tuning is robust [2505.11594].
- **Adaptive Strategies:** Modern kernels integrate similarity-based switches to guarantee that fully quantized paths are only executed when precision loss is benign, else a mixed or full-precision path is invoked.
- **Hardware Dependencies:** Full benefits are realized only on hardware supporting efficient INT8 or FP8 tensor-core matmuls and vector ops.

## 7. Practical Guidelines for Adopters

- Use per-block or vector-wise quantization with recalibration for each layer or attention head.
- For softmax, use block-aware input rescaling and quantized exponentiation in a format like HiF8.
- Enable outlier detection and high-precision decomposition (mixed-precision) for emergent features.
- For best inference performance, fuse quantization with kernel computation and select block sizes to match hardware warp/thread topology.
- In training, retain at least one key mat-mul in FP16 (e.g., $dO V^\top$) and restrict quantization to controlled blocks to avoid error accumulation.
- For transformer models with FlashAttention-style tiling, directly substitute with INT8-enabled kernels such as SageAttention or BAPS for significant acceleration without retraining.

References: [1711.03712], [2208.07339], [2410.02367], [2602.02071], [2505.11594]

Source: https://www.emergentmind.com/topics/8-bit-quantized-attention