--- title: Dynamic Outlier Quantization url: https://www.emergentmind.com/topics/dynamic-outlier-quantization type: topic --- # Dynamic Outlier Quantization Dynamic outlier quantization refers to a family of post-training or inference-time quantization methodologies designed to mitigate the severe accuracy and efficiency degradation that arises when large outlier values (“outliers”) are present in neural network weights or activations—especially as models are pushed to ultra-low-bit representations (e.g., W/A 4, 3, or even 2 bits). These approaches dynamically identify, redistribute, isolate, or otherwise neutralize the impact of outliers to support aggressive quantization without substantial loss of model accuracy or hardware efficiency. While specific implementations vary widely, fundamental elements include robust outlier detection (channel-wise, token-wise, or block-wise), mixed-precision or dual-path computation, block-wise reparameterization, and hardware-aware encoding. This article reviews foundational formulations, algorithmic variants, empirical tradeoffs, and technical details as presented in recent arXiv works. ## 1. Fundamental Problem: Outliers in Low-Bit Quantization The primary limitation of uniform low-bit quantization is that rare, large-magnitude values (“outliers”) dramatically expand the dynamic range, increasing quantization step size for all other values and causing unacceptable rounding or clipping error. In matrix multiplications, the presence of just a few outlier channels or tokens can render W4A4 quantization infeasible, especially in challenging LLM or vision transformer regimes [2403.06497]. Empirically, outliers may be present as: - Channel-wise: A small fraction of channels persistently exhibit much larger dynamic range than the rest [2604.12782]. - Token-wise: Certain tokens (e.g., “BOS”, delimiters, rare symbols) trigger extreme activations that ordinary per-channel or block-wise strategies do not address [2410.05265]. - Block-wise: In block floating-point schemes, any outlier within a block determines the shared exponent, collapsing the expressivity for the block’s remaining values [2403.20137]. - Temporal: In sequential models, which channel is the outlier may shift dynamically across time steps or samples [2503.10959]. Across these settings, outliers dominate quantization noise, accounting for 65% or more of total error under standard PTQ in transformers and diffusion models [2403.06497,2509.26436]. ## 2. Canonical Approaches: Outlier Identification and Isolation Dynamic outlier quantization begins with high-specificity detection mechanisms to flag outlier elements or groups. Typical procedures include: - **Statistical thresholding:** Outlier channels or elements are those whose maximum value exceeds a per-layer threshold, such as a fixed $\tau = \mu + \kappa \sigma$ or a quantile of the channel-wise maxima [2604.12782,1901.09504,2503.10959]. For example, OSC uses $T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$ for each layer [2604.12782]. - **Empirical clustering:** OSC and similar channel separation methods observe that outlier channels are “token-persistent”—the same channels act as outliers across the vast majority of sequence positions [2604.12782]. This motivates group-wise separation as in OSC, MUXQ, PrefixQuant, and block-permutation strategies in DuQuant [2406.01721]. - **Token-wise maxima:** PrefixQuant examines per-token activations and flags outlier tokens via $M_i / \mathrm{median}(M) > \eta$ (e.g., $\eta=64$) [2410.05265]. - **Activation-over-time:** In VMMs, outlier channels are recomputed at every time step using an adaptive threshold and periodic refresh logic [2503.10959]. - **High-dimensional unsupervised partitioning:** In QMC, a global outlier ratio $\rho$ is used to partition weights by magnitude at load time, enabling flexible adaptation to hardware constraints [2601.14549]. Isolation is then achieved through one of the following (sometimes composable) mechanisms: - Exclusion to a high-precision path (mixed-precision, e.g., OSC, OPAL, OWQ) [2604.12782,2409.05902,2306.02272]. - Block- or group-wise permutation to localize outliers and avoid their contamination of “clean” channels (DuQuant, BATQuant, block-FP K-sort) [2406.01721,2603.16590,2403.20137]. - Dynamic splitting (OCS, dynamic channel splitting) [1901.09504]. - Pre-filling strategies (PrefixQuant) that provoke all outliers in dedicated prefix tokens and remove them from the rest of the sequence [2410.05265]. ## 3. Algorithmic Frameworks: Block-Level, Channel-Wise, Token-Wise Leading implementations structure the dynamic outlier quantization process as follows: ### 3.1 Channel-Wise or Block-Wise Isolation Most methods rely on a static or dynamic partitioning of the tensor into blocks, within which the majority (“inliers”) are quantized with a small dynamic range, and a minority (“outliers”) are handled separately. For example, OSC zeroes out the dominant outlier channel in each group and routes it to a parallel high-precision GEMM path, incurring only $\sim$12.5% arithmetic overhead for $G=32$ [2604.12782]. MUXQ splits the activation matrix into a low-rank “body” (reduced via right-shift), and a small auxiliary matrix, both quantized to INT8, thus mitigating the effect of high-magnitude channels [2604.04701]. ### 3.2 Block Floating-Point Formats with Outlier Rearrangement Block floating-point (BFP) quantization schemes are highly sensitive to outliers, as any single large element in a block determines the exponent—collapsing precision for the rest. Dynamic outlier quantization statically permutes high-norm channels (using K-sort) into the same blocks so that the BFP scale is only influenced by other outliers, dramatically improving quantization accuracy [2403.20137]. ### 3.3 Mixed-Precision and Dual-Path Many methods allocate higher-precision (e.g., FP16 or 16-bit integer) storage and compute to outlier elements/channels/tokens, keeping the remainder in 4-, 3-, or even 2-bit representations. VMM-targeted approaches (OuroMamba-Quant) dynamically track outlier channels per time step and assign a higher bit-width to those channels, efficiently implemented as split GEMMs [2503.10959]. OWQ and ICQuant similarly split weight columns or elements by Hessian-weighted sensitivity, preserving a small set at full precision [2306.02272,2505.00850]. ### 3.4 Rotation, Permutation, and Distribution Strategies Rotation-based approaches design orthogonal or block-diagonal transformations to redistribute the energy of outlier activations across channels, reducing peak magnitude while maintaining invertibility. DuQuant composes two such block-wise rotations and a “zigzag” permutation, which distributes concentrated outliers across blocks, minimizing local variance and enabling much tighter quantization [2406.01721]. RotateKV optimizes Hadamard permutations (FWHT) in 2-bit KV quantization, with channel reordering and grouped-head logic to maximize hardware throughput and minimize interference with existing projections [2501.16383]. ### 3.5 Token-Wise Outlier Elimination PrefixQuant targets token-wise outliers by identifying and injecting outlier tokens into the prefix of the KV cache. After this static modification, all subsequent tokens are well-behaved, allowing global static quantization for the remainder of the sequence [2410.05265]. ### 3.6 OCS and Activation Splitting OCS (Outlier Channel Splitting) effectively halves the dynamic range by duplicating outlier channels, redistributing values toward the center of the quantization grid [1901.09504]. Dynamic variants preallocate spare channels and perform outlier detection on-the-fly to enable aggressive quantization for activations in distributionally-volatile deployment scenarios. ## 4. Empirical Trade-Offs and Technical Results Dynamic outlier quantization consistently delivers substantial empirical benefits in all major quantization benchmarks, closing much of the accuracy gap to full-precision at ultra-low-bits. Salient findings include: | Method | W/A bits | Model/Task | Accuracy Drop vs FP | Memory/Latency Benefits | Notable Statistics | | -------------- | ------------------ | --------------- | ------------------ | ---------------------------------- | --------------------------------------- | | OSC [2604.12782] | W4A4 | Qwen3-8B | 2.19 pp vs 6.09 (MXFP4) | 1.78$\times$ speedup | 12.5% HW overhead, Path B $1/G$ cycles | | BFP+K-sort [2403.20137] | BFP12 (K), BFP16 (Q) | Llama2-7B | 0.03 ppl vs FP16 | $2\times$ KV cache reduction | No runtime overhead | | OuroMamba-Quant [2503.10959] | W4A4 | Vim-S (ImageNet) | 5.7 pp vs FP32 (W4A4) | 2.36$\times$ speedup | Dynamic O($E\cdot N$) ops, $\ll$5% cost| | PrefixQuant [2410.05265] | W4A4KV4 | Llama3-8B | Up to 0.5–1.0 ppl (KV) | 2.8–3.3$\times$ faster quant kernel | Prefix detection $<$1 min on 70B model | | BATQuant [2603.16590] | W4A4KV16 | Qwen3-VL-8B | $\sim$3.6% of BF16 | Block-local GPK transformations | 96.43% recovery, outperform SOTA | | ICQuant [2505.00850] | 2.3b–2.44b | Llama3-70B | $<$1 ppl, up to 3$\times$ SOTA | 0.3 bits index coding; no FT needed | Halves quant range, minimal overhead | Fine-tuned methods (e.g., QuantTune) directly regularize the outlier deviation during post-training, demonstrating $>$12% accuracy gains at 8 bits, and up to $33.8\%$ improvement at 7 bits in vision transformer settings [2403.06497]. In diffusion model deployment, QuaRTZ leverages two-step quantization: first an 8-bit min–max pass isolates outliers, followed by leading zero suppression for aggressive 4-bit packing, achieving FID of 6.98 on FLUX.1-schnell without auxiliary branches [2509.26436]. ## 5. Hardware Integration and Efficiency A major motivation for dynamic outlier quantization is practical hardware realization: - **Separation of compute pathways** (OSC, OPAL) enables near-zero runtime branching and full utilization of native tensor-core and vector-mac units [2604.12782,2409.05902]. - **Efficient encoding/decoding**: Long-range index coding (ICQuant) and fixed-byte-aligned OVP quantization (OliVe) minimize memory bandwidth and area overhead while embedding outliers in situ [2505.00850,2304.07493]. - **Heterogeneous memory organizations** (QMC) dynamically partition model weights between low-noise MRAM for outliers and dense ReRAM for inliers, allowing hardware or system-level trade-off of latency, power, and fidelity in the field [2601.14549]. - **Block-local transforms**: BATQuant’s GPK decomposition matches accelerator block granularity, preventing cross-block contamination and minimizing parameter runtime footprint [2603.16590]. Empirically, methods such as OliVe report 3–5$\times$ GPU throughput and 2–4$\times$ energy savings over standard approaches at $<$1–2$\%$ accuracy degradation [2304.07493]. Prefixed-token approaches like PrefixQuant further eliminate quant kernel fusion barriers and enable significant prefill/decoding speedup for large LLMs [2410.05265]. ## 6. Limitations, Open Questions, and Directions Despite substantial progress, several open questions and limitations remain: - **Distributional shift and adaptation**: Approaches anchoring outlier indices in offline calibration (OSC, PrefixQuant, K-sort) are vulnerable to domain shift or unseen data. Methods that refresh outlier lists dynamically (OuroMamba-Quant) or permit runtime adjustment of outlier fraction ($\rho$ in QMC) are more resilient, but may raise hardware complexity or require monitoring infrastructure [2503.10959,2601.14549]. - **Per-token and temporal variability**: Token-wise outlier strategies rely on the assumption that outlier tokens are rare and fixed. In new domains or under subword tokenization drift, rare outlier tokens could still degrade performance if not classified in prefix sets [2410.05265]. - **Sequence-length and block granularity**: The effectiveness of block-wise transforms and separation depends on the alignment between hardware block size and the outlier spatial statistics. Overly coarse blocks can reintroduce mixing, reducing the achievable accuracy at fixed bit-width [2403.20137,2603.16590]. - **Complexity of dual/auxiliary computation paths**: While arithmetic and area overhead for split-path architectures is often small, aggressive scaling may run up against hardware architectural or scheduling limits, especially in resource-constrained embedded or edge use cases [2604.12782,2409.05902]. - **Generalization and extensibility**: Not all schemes can be directly applied to every architectural context (e.g., from LLMs to Mamba or DiT) without new calibration or permutation logic [2503.10959,2509.26436]. ## 7. Synthesis and Outlook Dynamic outlier quantization constitutes a set of methodologies that systematically decouple the error-inducing effects of rare, extreme-valued weights and activations in quantized neural networks. Algorithmic designs encompass channel and token identification, static and runtime block/permutation logic, mixed/hybrid-precision dual-path architectures, and problem-specific hardware encoding. Empirical evidence from recent arXiv works demonstrates that these approaches permit sub-4-bit quantization of LLMs and vision models with $<$1–2$\%$ accuracy loss, greatly reduced memory and bandwidth, and with overheads compatible with modern accelerator designs [2403.20137,2604.12782,2604.04701,2603.16590,2306.02272,2410.05265,2503.10959,2409.05902,2505.00850,2406.01721,2304.07493]. An ongoing research frontier involves compositionality—combining block-wise, channel-wise, and token-wise dynamic quantization with regularization-based approaches to achieve robust quantization under real distributional shift and on emerging hardware. Benchmarks now routinely require tuning across accuracy, memory, energy, and latency, mandating continued tight co-design of algorithms and hardware. Advances in dynamic outlier quantization will likely remain central as models and deployment scenarios diversify and quantization budgets become more aggressive. Source: https://www.emergentmind.com/topics/dynamic-outlier-quantization