OffQ: Taming Structured Outliers in LLM Quantization by Offsetting

Abstract: Low-bit quantization has been widely adopted to accelerate the inference of LLMs by significantly reducing computational cost and memory usage. However, activation outliers pose a major challenge to effective quantization, often leading to notable performance degradation. In this paper, we introduce OffQ, a method designed to mitigate activation outliers in low-bit quantization through a novel offsetting mechanism. Specifically, OffQ first identifies a low-dimensional outlier subspace in the activations using a proposed top-1 PCA, and then concentrates high-magnitude activations into 1 channel via rotation. OffQ then absorbs this concentrated outlier channel by converting its magnitude into a shared offset, thereby reducing the standard deviation of the activations. This offsetting strategy enables effective W4A4KV4 quantization of LLMs using deployment-friendly uniform-grid and uniform-precision quantization. Extensive experiments across diverse LLM architectures and benchmarks demonstrate that OffQ outperforms state-of-the-art baselines, consistently improving model accuracy while preserving low-bit efficiency.

• The paper introduces a novel post-training quantization method using top-1 PCA to isolate activation outliers, thereby reducing extreme activation variance.
• It applies group-wise Hadamard transforms to absorb outlier energy into uniform quantization, eliminating hardware complexities without runtime overhead.
• Empirical evaluations on Llama and Qwen models show significant perplexity improvements, outperforming existing rotation-based and mixed-precision methods.

OffQ: Taming Structured Outliers in LLM Quantization by Offsetting

Introduction and Motivation

Low-bit quantization is crucial for enabling inference of LLMs in resource-constrained environments. Despite advances in quantization algorithms, robust weight-activation-KV-cache (W4A4KV4) quantization remains problematic due to the presence of activation outliers. Such outliers, typically concentrated in a small number of channels, disproportionately expand the quantization dynamic range, leading to severe information loss and degraded performance at low precision. Prior approaches have focused variously on rotation-based transforms, mixed precision, or non-uniform quantization, each with key implementation caveats or hardware inefficiencies.

OffQ introduces a post-training quantization paradigm that systematically targets activation outliers by converting them into group-wise offsets, thus streamlining the distribution for uniform low-bit quantization. Central to OffQ is (1) the identification of a shared low-dimensional outlier subspace via a novel “top-1 PCA” and (2) the absorption of concentrated outlier energy as zero-point offsets via efficient orthogonal (Hadamard) transforms. OffQ forgoes complex kernel requirements, custom hardware, and high-precision fallback channels, instead offering hardware-friendly uniform quantization with strong empirical improvements.

Methodology

Outlier Subspace Identification: Top-1 PCA

OffQ leverages empirical observations from the LLM quantization literature that activation outliers exhibit pronounced low-rank structure, with high-norm tokens aligning along a single dominant direction [wang2025demystifying, bondarenko2021understanding]. OffQ formalizes this via a top-1 principal component analysis (PCA):

• For each sequence in a calibration set, the token with maximal $L^\infty$ norm is selected, targeting extreme activations.
• PCA is performed on this filtered token set, with the leading eigenvector capturing the major outlier direction.

By employing the top-1 selection instead of standard PCA over all tokens, OffQ ensures the extracted subspace is not diluted by the distribution of normal tokens. This allows more efficient outlier concentration, sharply reducing the variance contributed by normal activations.

Figure 1: Visualization of the outlier channel concentration on a Llama 3-8B activation before (left) and after (right) top-1 PCA rotation. Dominant activation energy is collapsed into a single channel.

Concentration and Offsetting via Orthogonal Transforms

After identifying the dominant outlier subspace, OffQ rotates activations to align these directions with the first $G$ channels. To mitigate the residual variance and absorb outlier energy, OffQ:

• Applies a group-wise Hadamard transform with the first row fixed to all-ones, yielding a group-wide constant offset proportional to outlier magnitude.
• This offset is absorbed via the zero-point of asymmetric quantization per group, neutralizing the ability of outliers to affect the representational fidelity of the remaining channels.
• Channels are assigned to outlier or non-outlier status via sorted permutation; non-outlier channels are grouped to minimize variance within each quantization group.

This sequential pipeline is highly efficient. The majority of rotation and permutation matrices are fused into weight matrices, introducing no runtime overhead after translation to deployment kernels.

Figure 2: OffQ pipeline: Top-1 PCA ($U_1$), grouping ($P_1$, $P_4$), random rotations ($R_2$), and group-wise Hadamard rotations ($H_1$, $H_3$, $H_4$). Most operations are pre-fused into model weights.

Quantization Process and Pipeline Integration

• For all linear layers, activations are right-multiplied by the joint rotation-mapping, permuted, and then group-wise Hadamard offset is applied.
• The corresponding inverse transforms are pre-applied or post-applied as needed to upstream and downstream matrices (e.g., weights, input/output embeddings).
• Weights are quantized using standard methods (here, GPTQ [frantar2023optq]), ensuring comparability with baselines.
• Per-group zero-point storage is minimal (4.25 effective bits/activation for group size 128 and 16-bit group metadata).

  Figure 3: Quantization performance (perplexity) vs. number of groups on Llama 3-8B. Finer group granularity leads to greater absorption of outlier channels and improved perplexity.

Experimental Evaluation

Extensive empirical evaluations are conducted on representative LLM families: Llama (2/3/3.2), Qwen 2.5 across a wide range of scales (1B–72B). Evaluation covers both linguistic generalization (WikiText perplexity) and zero-shot common-sense task accuracy (ARC, BoolQ, HellaSwag, PIQA, SIQA, WinoGrande, etc.).

Key findings:

• OffQ consistently outperforms all prior rotation-based and mixed-precision quantization methods across both model families and data regimes.
• On Llama 3-8B, OffQ achieves PPL 6.98 (0-shot 65.5) under W4A4KV4, compared with ResQ (PPL 7.1), KurTail (PPL 7.2), and QuaRot (PPL 7.8). On Qwen 2.5-7B, OffQ yields PPL 7.66 (0-shot 66.2), outperforming ResQ (PPL 8.2), QuaRot (PPL 4036).
• OffQ's performance gains are largest in small- to medium-scale LLMs and models with severe outlier structure (e.g., Qwen 2.5, which previously broke existing W4A4KV4 pipelines).
• Ablation studies confirm critical contributions from the top-1 PCA, sorted grouping, and the explicit Hadamard construction for group-wise offsetting.
• Empirical scaling analysis reveals diminishing returns with smaller group size, providing actionable guidelines for deployment-time parameter selection.

Theoretical and Practical Implications

OffQ hypothesizes and confirms a fundamental duality between activation outlier structure and quantization capacity limits. By reducing effective distributional variance via systematic subspace manipulation and group-wise offset absorption, OffQ allows uniform low-precision quantization without significant representational loss or hardware complexity.

Implications and future work:

• Hardware enablement: OffQ's approach is congruent with modern low-bit-optimized hardware (e.g., Tensor Cores, block-wise quantization), requiring only minimal additional metadata (~0.25 bits/activation).
• Model scalability: OffQ is robust under scale, outperforming previous bests even as model activations become more skewed.
• Composability: OffQ's transform is orthogonal to (and can be combined with) recent advances in non-uniform and learned rotation techniques.
• Generalizability: The offsetting mechanism is readily extensible to other architectures with structured outliers (e.g., ViTs), and to formats requiring block-based quantization.

Conclusion

OffQ establishes a new state-of-the-art approach for W4A4KV4 quantization in LLMs by introducing an activation offsetting mechanism driven by top-1 PCA subspace analysis. The method directly addresses the core challenge of structured activation outliers, yielding robust performance improvements over existing rotation-based and mixed-precision methods, especially on outlier-heavy or resource-constrained deployments. OffQ’s design offers immediate practicality for both academic research and industry deployment, and suggests a broader paradigm of using outlier-to-offset mappings as a foundation for advanced quantization strategies in neural architectures.