Dynamic Outlier Quantization

• Dynamic outlier quantization is a methodology that identifies and neutralizes extreme neural network activations to sustain accuracy in low-bit settings.
• It utilizes statistical thresholding, empirical clustering, and mixed-precision dual-path strategies to manage outliers at channel, token, or block levels.
• Empirical evidence demonstrates significant accuracy retention, reduced memory overhead, and enhanced hardware throughput in modern architectures.

Dynamic outlier quantization refers to a family of post-training or inference-time quantization methodologies designed to mitigate the severe accuracy and efficiency degradation that arises when large outlier values (“outliers”) are present in neural network weights or activations—especially as models are pushed to ultra-low-bit representations (e.g., W/A 4, 3, or even 2 bits). These approaches dynamically identify, redistribute, isolate, or otherwise neutralize the impact of outliers to support aggressive quantization without substantial loss of model accuracy or hardware efficiency. While specific implementations vary widely, fundamental elements include robust outlier detection (channel-wise, token-wise, or block-wise), mixed-precision or dual-path computation, block-wise reparameterization, and hardware-aware encoding. This article reviews foundational formulations, algorithmic variants, empirical tradeoffs, and technical details as presented in recent arXiv works.

1. Fundamental Problem: Outliers in Low-Bit Quantization

The primary limitation of uniform low-bit quantization is that rare, large-magnitude values (“outliers”) dramatically expand the dynamic range, increasing quantization step size for all other values and causing unacceptable rounding or clipping error. In matrix multiplications, the presence of just a few outlier channels or tokens can render W4A4 quantization infeasible, especially in challenging LLM or vision transformer regimes (Chen et al., 2024). Empirically, outliers may be present as:

• Channel-wise: A small fraction of channels persistently exhibit much larger dynamic range than the rest (Zhang et al., 14 Apr 2026).
• Token-wise: Certain tokens (e.g., “BOS”, delimiters, rare symbols) trigger extreme activations that ordinary per-channel or block-wise strategies do not address (Chen et al., 2024).
• Block-wise: In block floating-point schemes, any outlier within a block determines the shared exponent, collapsing the expressivity for the block’s remaining values (Trukhanov et al., 2024).
• Temporal: In sequential models, which channel is the outlier may shift dynamically across time steps or samples (Ramachandran et al., 13 Mar 2025).

Across these settings, outliers dominate quantization noise, accounting for 65% or more of total error under standard PTQ in transformers and diffusion models (Chen et al., 2024Kim et al., 30 Sep 2025).

2. Canonical Approaches: Outlier Identification and Isolation

Dynamic outlier quantization begins with high-specificity detection mechanisms to flag outlier elements or groups. Typical procedures include:

• Statistical thresholding: Outlier channels or elements are those whose maximum value exceeds a per-layer threshold, such as a fixed $\tau = \mu + \kappa \sigma$ or a quantile of the channel-wise maxima (Zhang et al., 14 Apr 2026Zhao et al., 2019Ramachandran et al., 13 Mar 2025). For example, OSC uses $T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$ for each layer (Zhang et al., 14 Apr 2026).
• Empirical clustering: OSC and similar channel separation methods observe that outlier channels are “token-persistent”—the same channels act as outliers across the vast majority of sequence positions (Zhang et al., 14 Apr 2026). This motivates group-wise separation as in OSC, MUXQ, PrefixQuant, and block-permutation strategies in DuQuant (Lin et al., 2024).
• Token-wise maxima: PrefixQuant examines per-token activations and flags outlier tokens via $M_i / \mathrm{median}(M) > \eta$ (e.g., $\eta=64$) (Chen et al., 2024).
• Activation-over-time: In VMMs, outlier channels are recomputed at every time step using an adaptive threshold and periodic refresh logic (Ramachandran et al., 13 Mar 2025).
• High-dimensional unsupervised partitioning: In QMC, a global outlier ratio $\rho$ is used to partition weights by magnitude at load time, enabling flexible adaptation to hardware constraints (Pandey et al., 21 Jan 2026).

Isolation is then achieved through one of the following (sometimes composable) mechanisms:

• Exclusion to a high-precision path (mixed-precision, e.g., OSC, OPAL, OWQ) (Zhang et al., 14 Apr 2026Koo et al., 2024Lee et al., 2023).
• Block- or group-wise permutation to localize outliers and avoid their contamination of “clean” channels (DuQuant, BATQuant, block-FP K-sort) (Lin et al., 2024Li et al., 17 Mar 2026Trukhanov et al., 2024).
• Dynamic splitting (OCS, dynamic channel splitting) (Zhao et al., 2019).
• Pre-filling strategies (PrefixQuant) that provoke all outliers in dedicated prefix tokens and remove them from the rest of the sequence (Chen et al., 2024).

3. Algorithmic Frameworks: Block-Level, Channel-Wise, Token-Wise

Leading implementations structure the dynamic outlier quantization process as follows:

3.1 Channel-Wise or Block-Wise Isolation

Most methods rely on a static or dynamic partitioning of the tensor into blocks, within which the majority (“inliers”) are quantized with a small dynamic range, and a minority (“outliers”) are handled separately. For example, OSC zeroes out the dominant outlier channel in each group and routes it to a parallel high-precision GEMM path, incurring only $\sim$12.5% arithmetic overhead for $G=32$ (Zhang et al., 14 Apr 2026). MUXQ splits the activation matrix into a low-rank “body” (reduced via right-shift), and a small auxiliary matrix, both quantized to INT8, thus mitigating the effect of high-magnitude channels (Lee et al., 6 Apr 2026).

3.2 Block Floating-Point Formats with Outlier Rearrangement

Block floating-point (BFP) quantization schemes are highly sensitive to outliers, as any single large element in a block determines the exponent—collapsing precision for the rest. Dynamic outlier quantization statically permutes high-norm channels (using K-sort) into the same blocks so that the BFP scale is only influenced by other outliers, dramatically improving quantization accuracy (Trukhanov et al., 2024).

3.3 Mixed-Precision and Dual-Path

Many methods allocate higher-precision (e.g., FP16 or 16-bit integer) storage and compute to outlier elements/channels/tokens, keeping the remainder in 4-, 3-, or even 2-bit representations. VMM-targeted approaches (OuroMamba-Quant) dynamically track outlier channels per time step and assign a higher bit-width to those channels, efficiently implemented as split GEMMs (Ramachandran et al., 13 Mar 2025). OWQ and ICQuant similarly split weight columns or elements by Hessian-weighted sensitivity, preserving a small set at full precision (Lee et al., 2023Li et al., 1 May 2025).

3.4 Rotation, Permutation, and Distribution Strategies

Rotation-based approaches design orthogonal or block-diagonal transformations to redistribute the energy of outlier activations across channels, reducing peak magnitude while maintaining invertibility. DuQuant composes two such block-wise rotations and a “zigzag” permutation, which distributes concentrated outliers across blocks, minimizing local variance and enabling much tighter quantization (Lin et al., 2024). RotateKV optimizes Hadamard permutations (FWHT) in 2-bit KV quantization, with channel reordering and grouped-head logic to maximize hardware throughput and minimize interference with existing projections (Su et al., 25 Jan 2025).

3.5 Token-Wise Outlier Elimination

PrefixQuant targets token-wise outliers by identifying and injecting outlier tokens into the prefix of the KV cache. After this static modification, all subsequent tokens are well-behaved, allowing global static quantization for the remainder of the sequence (Chen et al., 2024).

3.6 OCS and Activation Splitting

OCS (Outlier Channel Splitting) effectively halves the dynamic range by duplicating outlier channels, redistributing values toward the center of the quantization grid (Zhao et al., 2019). Dynamic variants preallocate spare channels and perform outlier detection on-the-fly to enable aggressive quantization for activations in distributionally-volatile deployment scenarios.

4. Empirical Trade-Offs and Technical Results

Dynamic outlier quantization consistently delivers substantial empirical benefits in all major quantization benchmarks, closing much of the accuracy gap to full-precision at ultra-low-bits. Salient findings include:

Method	W/A bits	Model/Task	Accuracy Drop vs FP	Memory/Latency Benefits	Notable Statistics
OSC (Zhang et al., 14 Apr 2026)	W4A4	Qwen3-8B	2.19 pp vs 6.09 (MXFP4)	1.78$\times$ speedup	12.5% HW overhead, Path B $1/G$ cycles
BFP+K-sort (Trukhanov et al., 2024)	BFP12 (K), BFP16 (Q)	Llama2-7B	0.03 ppl vs FP16	$2\times$ KV cache reduction	No runtime overhead
OuroMamba-Quant (Ramachandran et al., 13 Mar 2025)	W4A4	Vim-S (ImageNet)	5.7 pp vs FP32 (W4A4)	2.36$T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$0 speedup	Dynamic O($T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$1) ops, $T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$25% cost
PrefixQuant (Chen et al., 2024)	W4A4KV4	Llama3-8B	Up to 0.5–1.0 ppl (KV)	2.8–3.3$T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$3 faster quant kernel	Prefix detection $T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$41 min on 70B model
BATQuant (Li et al., 17 Mar 2026)	W4A4KV16	Qwen3-VL-8B	$T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$53.6% of BF16	Block-local GPK transformations	96.43% recovery, outperform SOTA
ICQuant (Li et al., 1 May 2025)	2.3b–2.44b	Llama3-70B	$T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$61 ppl, up to 3$T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$7 SOTA	0.3 bits index coding; no FT needed	Halves quant range, minimal overhead

Fine-tuned methods (e.g., QuantTune) directly regularize the outlier deviation during post-training, demonstrating $T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$812% accuracy gains at 8 bits, and up to $T_\ell = 5\,\mathbb{E}[|X_{i,j}|]$9 improvement at 7 bits in vision transformer settings (Chen et al., 2024).

In diffusion model deployment, QuaRTZ leverages two-step quantization: first an 8-bit min–max pass isolates outliers, followed by leading zero suppression for aggressive 4-bit packing, achieving FID of 6.98 on FLUX.1-schnell without auxiliary branches (Kim et al., 30 Sep 2025).

5. Hardware Integration and Efficiency

A major motivation for dynamic outlier quantization is practical hardware realization:

• Separation of compute pathways (OSC, OPAL) enables near-zero runtime branching and full utilization of native tensor-core and vector-mac units (Zhang et al., 14 Apr 2026Koo et al., 2024).
• Efficient encoding/decoding: Long-range index coding (ICQuant) and fixed-byte-aligned OVP quantization (OliVe) minimize memory bandwidth and area overhead while embedding outliers in situ (Li et al., 1 May 2025Guo et al., 2023).
• Heterogeneous memory organizations (QMC) dynamically partition model weights between low-noise MRAM for outliers and dense ReRAM for inliers, allowing hardware or system-level trade-off of latency, power, and fidelity in the field (Pandey et al., 21 Jan 2026).
• Block-local transforms: BATQuant’s GPK decomposition matches accelerator block granularity, preventing cross-block contamination and minimizing parameter runtime footprint (Li et al., 17 Mar 2026).

Empirically, methods such as OliVe report 3–5$M_i / \mathrm{median}(M) > \eta$0 GPU throughput and 2–4$M_i / \mathrm{median}(M) > \eta$1 energy savings over standard approaches at $M_i / \mathrm{median}(M) > \eta$21–2$M_i / \mathrm{median}(M) > \eta$3 accuracy degradation (Guo et al., 2023). Prefixed-token approaches like PrefixQuant further eliminate quant kernel fusion barriers and enable significant prefill/decoding speedup for large LLMs (Chen et al., 2024).

6. Limitations, Open Questions, and Directions

Despite substantial progress, several open questions and limitations remain:

• Distributional shift and adaptation: Approaches anchoring outlier indices in offline calibration (OSC, PrefixQuant, K-sort) are vulnerable to domain shift or unseen data. Methods that refresh outlier lists dynamically (OuroMamba-Quant) or permit runtime adjustment of outlier fraction ($M_i / \mathrm{median}(M) > \eta$4 in QMC) are more resilient, but may raise hardware complexity or require monitoring infrastructure (Ramachandran et al., 13 Mar 2025Pandey et al., 21 Jan 2026).
• Per-token and temporal variability: Token-wise outlier strategies rely on the assumption that outlier tokens are rare and fixed. In new domains or under subword tokenization drift, rare outlier tokens could still degrade performance if not classified in prefix sets (Chen et al., 2024).
• Sequence-length and block granularity: The effectiveness of block-wise transforms and separation depends on the alignment between hardware block size and the outlier spatial statistics. Overly coarse blocks can reintroduce mixing, reducing the achievable accuracy at fixed bit-width (Trukhanov et al., 2024Li et al., 17 Mar 2026).
• Complexity of dual/auxiliary computation paths: While arithmetic and area overhead for split-path architectures is often small, aggressive scaling may run up against hardware architectural or scheduling limits, especially in resource-constrained embedded or edge use cases (Zhang et al., 14 Apr 2026Koo et al., 2024).
• Generalization and extensibility: Not all schemes can be directly applied to every architectural context (e.g., from LLMs to Mamba or DiT) without new calibration or permutation logic (Ramachandran et al., 13 Mar 2025Kim et al., 30 Sep 2025).

7. Synthesis and Outlook

Dynamic outlier quantization constitutes a set of methodologies that systematically decouple the error-inducing effects of rare, extreme-valued weights and activations in quantized neural networks. Algorithmic designs encompass channel and token identification, static and runtime block/permutation logic, mixed/hybrid-precision dual-path architectures, and problem-specific hardware encoding. Empirical evidence from recent arXiv works demonstrates that these approaches permit sub-4-bit quantization of LLMs and vision models with $M_i / \mathrm{median}(M) > \eta$51–2$M_i / \mathrm{median}(M) > \eta$6 accuracy loss, greatly reduced memory and bandwidth, and with overheads compatible with modern accelerator designs (Trukhanov et al., 2024Zhang et al., 14 Apr 2026Lee et al., 6 Apr 2026Li et al., 17 Mar 2026Lee et al., 2023Chen et al., 2024Ramachandran et al., 13 Mar 2025Koo et al., 2024Li et al., 1 May 2025Lin et al., 2024Guo et al., 2023).

An ongoing research frontier involves compositionality—combining block-wise, channel-wise, and token-wise dynamic quantization with regularization-based approaches to achieve robust quantization under real distributional shift and on emerging hardware. Benchmarks now routinely require tuning across accuracy, memory, energy, and latency, mandating continued tight co-design of algorithms and hardware. Advances in dynamic outlier quantization will likely remain central as models and deployment scenarios diversify and quantization budgets become more aggressive.