Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs

Abstract: LLMs achieve strong performance but incur high deployment costs, motivating extremely low-bit but lossy quantization. Existing quantization algorithms mainly focus on improving the numerical accuracy of forward computation to eliminate performance degradation. In this paper, we show that extremely quantized LLMs suffer from systematic smoothness degradation beyond numerical precision loss. Through a smoothness proxy, we observe that such degradation becomes increasingly severe as the quantization bit-width decreases. Furthermore, based on sequence neighborhood modeling, we find that quantized models exhibit a rapid reduction of effective token candidates within the prediction neighborhood, which directly leads to a sparser decoding tree and degraded generation quality. To validate it, we introduce a simple smoothness-preserving principle in both post-training quantization and quantization-aware training, and demonstrate that preserving smoothness brings additional gains beyond numerical accuracy. The core goal of this paper is to highlight smoothness preservation as an important design consideration for future extreme quantization methods. Code is available at https://github.com/xuyuzhuang11/FINE.

• The paper demonstrates that solely minimizing fitting error in extremely quantized LLMs fails to preserve output stability and token diversity.
• The paper introduces learnable gradient preservation and loss of gradient regularization techniques to maintain smoothness under extreme bit-width reductions.
• The paper provides both empirical and theoretical analyses that reveal a trade-off between forward accuracy and smoothness, with significant implications for generative performance.

Smoothness as a Crucial Objective in Extremely Quantized LLMs

Introduction

The paper "Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs" (2605.08894) tackles a critical limitation in the domain of extreme LLM quantization. While previous research has predominantly targeted the minimization of forward numerical error to retain pre-quantization performance, this study exposes the inadequacy of a sole focus on lossless fitting. Instead, it positions "smoothness"—operationalized as the stability of a model's output to input and hidden state perturbations—as a parallel, underexplored axis whose degradation in low-bit quantization leads to substantial utility loss, especially in text generation settings. The authors substantiate this claim both empirically and theoretically, introduce practical smoothness-preserving mechanisms, and provide guidance for future algorithmic design.

Quantization-Induced Smoothness Degradation

The empirical section establishes, via a smoothness proxy grounded in input gradients, that reduction in quantization bit-width (e.g., to 2- or 1.58-bit representations) induces substantial smoothness decay in LLMs. This is visualized through smoothness score distributions over LLaMA-2-7B quantized using GPTQ, where higher scores denote rougher function landscapes and increased sensitivity to minute input changes.

Figure 1: Smoothness score distributions of GPTQ-quantized LLaMA-2-7B under different bit-widths; lower bit-widths show drastically worsened smoothness.

Notably, classical metrics such as perplexity and local reconstruction error fail to capture the severity of this phenomenon, which manifests as an increased norm of input and hidden state gradients under extreme quantization. The result is greater proneness to erratic output and ranking shifts, even among high-quality candidate tokens.

Theoretical Framework: Sequence Neighborhood Collapse

Beyond empirical demonstrations, the paper introduces a formal analysis based on sequence neighborhood modeling. In generation, LLMs operate by ranking next-token candidates per context, which can be seen as perturbation directions in sequence space. The key finding is that quantized models rapidly reduce the effective number of high-quality next-token candidates—a phenomenon the authors term "neighborhood collapse." This is quantitatively tracked by "reverse perplexity" (rPPL), measuring the perplexity of tokens suggested by the quantized model when rescored by the original full-precision model.

Figure 3: Schematic definition of the local sequence neighborhood $N_\delta(w_{1:T})$ underlying rPPL.

The empirical rPPL curves for various quantization strategies (GPTQ, AQLM) reveal that as bit-width decreases, the rPPL increases dramatically beyond the top-1 prediction. This collapse is not reflected in traditional perplexity but leads to much sparser decoding trees, with far fewer effective generation paths available, consistently compromising generative diversity and quality.

Figure 2: Decoding tree for GPTQ-quantized LLaMA-2-7B; the tree becomes sparser as bit-width decreases.

Smoothness Preservation Strategies

To directly counteract quantization-induced smoothness degradation, the authors propose two algorithmic interventions:

Learnable Gradient Preservation (LGP) for PTQ

LGP augments standard post-training quantization objectives (focused on activation reconstruction error) with a gradient preservation term, explicitly encouraging alignment between the backward-pass gradients of the quantized and full-precision models. This is motivated by the observation that forward accuracy and backward smoothness are orthogonal in the quantized weight space.

Figure 4: Illustration of the row-wise forward pass and the column-wise backward pass—showing that smoothness must be preserved in distinct directions.

Through integration into OmniQuant, LGP produces 2-bit quantized models that outperform standard GPTQ and OmniQuant baselines on a broad set of downstream tasks, yielding consistent improvements in both perplexity and zero-shot accuracy, despite no explicit optimization of fitting accuracy alone.

Loss of Gradient Regularization (LGR) for QAT

Quantization-aware training is similarly enhanced with gradient regularization, but due to the presence of a "gradient ridge" at the 0-th (embedding) layer, LGR is applied to the first hidden state's input gradient. This aligns with empirical findings that quantization disproportionately amplifies intermediate hidden state gradients, not just the initial input gradients.

Disentangling Accuracy and Smoothness

A critical scientific contribution lies in disentangling forward accuracy and smoothness as partially independent axes in quantized model performance. Experimental manipulation of quantized weight solutions (by interpolating between those optimized for accuracy and those for backward smoothness) demonstrates that at modest quantization levels (4-bit), accuracy and smoothness optima closely coincide. In contrast, as bit-width drops, the trade-off becomes sharply anisotropic, and backward smoothness is more sensitive to precision loss than forward accuracy.

Figure 7: Pareto frontiers shift rapidly as quantization bit-width decreases, making joint optimization of forward and backward behavior increasingly difficult.

It is shown that the joint constraint does not nullify feasible solutions, thanks to the low effective rank (and hence high nullity) of LLM inputs and gradients, but the solution space does narrow substantially.

Mechanistic Insights: Why Smoothness Matters

Theoretical sequence analysis and ablations reveal that smoothness regularization does not merely shift probability mass, but crucially improves the relative ranking of high-quality tokens in the local neighborhood of the decoding tree. In downstream tasks such as multiple-choice QA, this effect can yield accuracy improvements even when the change in absolute probabilities is small. Thus, smoothness acts as a complementary, not replaceable, augmentation to classical fitting-focused quantization.

Figure 5: Comparison of outcomes with fitting vs. smoothness optimization, showing ideal ranking behavior critical for multi-option tasks.

Layer Normalization and Training Dynamics

LayerNorm is identified as essential not only for forward stability but also for smoothing intermediate gradients, highlighting the necessity of normalization modules in sub-2-bit training regimes. The paper also explores the origins and pervasiveness of the embedding-layer "gradient ridge," relating it to the sparsity and lack of semantic structure in word embeddings.

Implications and Future Directions

This study fundamentally recasts the extreme quantization challenge as a dual-objective problem: both forward-fitting accuracy and smoothness must be preserved for high-fidelity generative modeling under severe bit budgets. Future algorithm development should focus on joint optimization solvers that achieve optimal trade-offs, potentially leveraging Pareto-aware or multi-objective optimization frameworks. Additionally, integration of backward-gradient importance metrics into residual quantization and better understanding of the gradient ridge's origins may catalyze advances in neural compression robustness, interpretability, and theoretical analysis.

Conclusion

The work decisively invalidates the sufficiency of sole forward fitting for retaining LLM capability in the extreme quantization regime. Through a combination of empirical, theoretical, and optimization-based evidence, it demonstrates that preserving output smoothness is a principal lever for maintaining token diversity, robustness, and downstream task accuracy. The proposed LGP and LGR methods, although basic, provide effective baselines and actionable guidance for future compression methods—solidifying smoothness as a central criterion for next-generation LLM deployment under stringent resource constraints.