OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization

Abstract: LLMs have demonstrated remarkable capabilities. However, their massive parameter scale leads to significant resource consumption and latency during inference. Post-training weight-only quantization offers a promising solution by reducing model size and accelerating token generation through alleviating the memory-bound issue. Nevertheless, the presence of inherent systematic outliers in weights continues to be a major obstacle. While existing methods, such as scaling and rotation, attempt to address this issue, the performance remains unsatisfactory. In this paper, we propose Outlier Self-Absorption Quantization (OSAQ), which performs additive weight suppression guided by the second-order low-rank property for low-bit weight-only quantization of LLMs. Specifically, we observe that the Hessian exhibits low-rank consistency across different inputs, with certain directions consistently showing vanishing curvature. Leveraging this property, we identify a stable null space of the Hessian and then construct an additive weight transformation by linearly combining the vectors within this null space, thereby suppressing weight outliers without affecting the task loss. This additive transformation can be absorbed into the weights offline, requiring no inter-layer transformations and introducing no inference overhead. Moreover, the construction is efficiently achieved by a closed-form solution, without resource-intensive training or iterative procedures. Extensive experiments demonstrate that OSAQ effectively suppresses outliers and enhances low-bit quantization performance. For instance, in 2-bit quantization, OSAQ, when integrated with GPTQ, achieves over 40% lower perplexity compared to vanilla GPTQ.

• The paper presents an additive weight adjustment method using the Hessian's null space to mitigate outlier effects in low-bit LLM quantization.
• It employs a non-iterative, closed-form solution that achieves over 40% lower perplexity in ultra-low-bit regimes compared to traditional methods.
• The approach complements existing operator scaling techniques, ensuring minimal runtime overhead and preserving full-precision accuracy.

Outlier Self-Absorption Quantization: A Formal Analysis

Introduction

The exponential growth in parameter count and resource requirements in LLMs imposes significant challenges for deployment in real-world settings, particularly those constrained by low-latency and cost requirements. While post-training quantization (PTQ) has been a critical tool for model compression and acceleration, the presence of systematic outliers in LLM weight distributions fundamentally undermines low-bit quantization efficacy. Traditional solutions rooted in multiplicative inter-layer operations (e.g., scaling, rotation) are only partially effective and plateau in their ability to suppress the negative impact of these outliers. The paper "OSAQ: Outlier Self-Absorption for Accurate Low-bit LLM Quantization" (2605.04738) addresses these limitations via an orthogonal strategy: additive weight transformations informed by the low-rank, input-consistent null space of the Hessian.

Methodology

Low-Rank Hessian Analysis and Null Space Extraction

A central empirical finding of the paper is that, for LLMs, the per-layer Hessian of the task loss with respect to weights exhibits pronounced low-rank structure across a wide range of input distributions. The null space associated with vanishing Hessian eigenvalues is stable across calibration and inference distributions, as quantified by high principal angle cosine similarity. This stability enables robust identification of perturbation directions along which the second-order loss remains invariant under weight transformations.

Additive Outlier Suppression

Building on this, OSAQ supplements traditional multiplicative outlier suppression schemes with an additive paradigm. For each layer, a set of null space vectors is extracted via eigendecomposition of the Hessian. An additive reparameterization is formed as a linear combination of these vectors, with combination coefficients optimized—using a closed-form solution based on a differentiable $\ell_\infty$-norm surrogate via softmax-$l_2$ approximation—to directly minimize the range of the weight values. This suppresses outlier magnitudes without incurring either cross-layer dependency or any first/second-order change in the loss landscape.

Algorithmic Efficiency

The entire OSAQ routine is non-iterative and admits a closed-form solution for the optimal additive adjustment matrix, avoiding the computation overhead of retraining or online search. As a consequence, OSAQ introduces only marginal quantization runtime overhead and is entirely absorbed into the weight tensors offline, ensuring zero inference latency penalty.

Experimental Results

The paper reports comprehensive benchmarking across LLaMA2 (7B/13B/70B), LLaMA3 (8B/70B), and very large instruction-tuned models (123B, 405B) in both language generation (perplexity on WikiText2/C4) and zero-shot QA (PIQA, ARC, WinoGrande) as well as the MMLU multitask and MT-Bench multi-turn dialogue benchmarks.

Strong Numerical Results

• 2-bit Quantization: OSAQ+GPTQ achieves over 40% lower perplexity compared to vanilla GPTQ, e.g., reducing WikiText2 perplexity from 36.8 to 21.2 on LLaMA2-7B.
• 3/4-bit Regime: Consistent improvement in perplexity and accuracy across all evaluated models and tasks, with robustness to hyperparameter selection and minimal degradation in FP16 accuracy after the transformation.
• When combined with state-of-the-art baselines (GPTQ, AWQ, QuIP), OSAQ delivers additive gains, demonstrating compatibility and synergy.
• For instruction-tuned models exceeding 100B parameters, OSAQ remains effective, yielding further improvement over strong baselines.
• The stability of the Hessian null space under calibration/inference distribution mismatch and small calibration set sizes underpins this generalization.

Analysis of the Additive Mechanism

The ablation studies confirm that the additive transformation produces negligible changes in full-precision model accuracy while dramatically benefiting low-bit quantization. The differentiable softmax-$l_2$ surrogate for $\ell_\infty$-norm minimization outperforms direct $\ell_2$ minimization, particularly in outlier suppression—validated by weight distribution visualizations before and after OSAQ.

Inference and Runtime Benchmarks

OSAQ does not alter the decoding path or introduce extra computations at inference, preserving the expected acceleration ratios from quantization observed on modern hardware.

Theoretical and Practical Implications

The formal exploitation of input-consistent, low-curvature directions highlights the presence of actionable, loss-invariant structure in modern LLM parameter spaces. OSAQ's design is complementary to operator scaling and rotation-based strategies and closes an important gap in quantization methodology: addressing outlier suppression without inducing cross-layer dependencies or retraining, and in a way that is agnostic to downstream quantization schemes.

Practically, OSAQ advances the state of the art for ultra-low-bit (2/3-bit) LLM quantization, significantly reducing the memory and computational bottleneck without accuracy compromise, and is compatible with commonly adopted quantization baselines.

Future Directions

The implications of model- and data-agnostic Hessian null space stability open further research into loss-invariant transformations with broader application, such as structured pruning, automated architecture search, and more efficient quantized training paradigms. Incorporating finer-grained Hessian analysis, adaptive null space selection, and combination with activation-side quantization strategies (motivated by recent works such as QuaRot and SpinQuant) are natural extensions. OSAQ-type additive transformations may also facilitate robust deployment to rapidly evolving hardware backends with dynamic precision requirements.

Conclusion

OSAQ introduces a formal, loss-invariant additive paradigm for low-bit weight-only quantization in LLMs by leveraging the input-consistent, low-curvature null space of the Hessian. It achieves significant improvements over leading PTQ methods, particularly in the 2-bit and 3-bit regimes, without adverse runtime or inference impacts. The work represents a principled advance in model compression and sets a new benchmark for practical, accurate, low-bit quantization of large-scale foundation models.