SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models

Abstract: LLMs have shown remarkable performance in various domains, but they are constrained by massive computational and storage costs. Quantization, an effective technique for compressing models to fit resource-limited devices while preserving generative quality, encompasses two primary methods: quantization aware training (QAT) and post-training quantization (PTQ). QAT involves additional retraining or fine-tuning, thus inevitably resulting in high training cost and making it unsuitable for LLMs. Consequently, PTQ has become the research hotspot in recent quantization methods. However, existing PTQ methods usually rely on various complex computation procedures and suffer from considerable performance degradation under low-bit quantization settings. To alleviate the above issues, we propose a simple and effective post-training quantization paradigm for LLMs, named SEPTQ. Specifically, SEPTQ first calculates the importance score for each element in the weight matrix and determines the quantization locations in a static global manner. Then it utilizes the mask matrix which represents the important locations to quantize and update the associated weights column-by-column until the appropriate quantized weight matrix is obtained. Compared with previous methods, SEPTQ simplifies the post-training quantization procedure into only two steps, and considers the effectiveness and efficiency simultaneously. Experimental results on various datasets across a suite of models ranging from millions to billions in different quantization bit-levels demonstrate that SEPTQ significantly outperforms other strong baselines, especially in low-bit quantization scenarios.

• The paper presents SEPTQ, a novel two-step paradigm that statically preserves top-weight importance to effectively minimize error in ultra-low-bit quantization.
• It leverages global importance scoring and error-compensated weight updates to achieve 4–28% accuracy improvements over dynamic methods in 2-bit regimes.
• Empirical validation on OPT, LLaMA, and LLaMA2 models shows that SEPTQ maintains high performance by retaining as little as 0.1% of parameters, enabling efficient edge deployment.

SEPTQ: A Simple and Effective Paradigm for Post-Training Quantization of LLMs

Motivation and Context

The parameter explosion in LLMs impedes cost-effective deployment, especially on resource-constrained devices. Post-training quantization (PTQ) allows compression of pretrained models by reducing the precision of their weights, yet existing PTQ approaches either incur substantial performance degradation at aggressive quantization levels or introduce algorithmic complexity and prohibitive computational cost. SEPTQ ("Simple and Effective Post-Training Quantization paradigm") addresses both the efficacy and efficiency gaps in ultra-low-bit quantization of LLMs, specifically in 2/3/4-bit regimes, by introducing a streamlined two-step procedure for weight quantization.

Overview of the SEPTQ Paradigm

SEPTQ achieves PTQ by combining static global importance scoring for weight selection with efficient error-compensated quantization, guided by a mask matrix $\mathbf{M}$. The paradigm operates as follows:

1. Quantization Location Determination: SEPTQ statically computes an importance score for each weight in a linear layer using a closed-form criterion derived from the expected squared output error after quantization. The top $p\%$ of weights by score are preserved in full-precision, while remaining weights are quantized.
2. Weight Quantization and Compensation: The quantized and preserved weights are updated column-wise; for quantized locations, explicit error propagation updates the remainder of each weight row to optimally compensate for quantization loss, following a Lagrangian formulation.

This approach is distinct in its static (non-iterative) global selection of weights to preserve, contrasting with prior dynamic or local selection schemes. Such statically-determined masks significantly accelerate quantization and reduce computational overhead, while also retaining an outsized fraction of model fidelity with minimal uncompressed weights.

Quantization Location Selection: Analysis and Visualization

Empirical analysis demonstrates a pronounced heavy-tailed distribution in importance scores across LLM weight matrices: a minuscule subset of weights dominates error contribution under quantization, while the majority have negligible impact. This is particularly accentuated under more aggressive (2-bit) regimes, driving the rationale for static global selection of "important" weights.

Figure 1: Distribution of importance scores in 4-bit quantization; most weights have low importance, with a long tail of high-importance weights.

The contrast between 4-bit and 2-bit quantization illustrates that as bit-width decreases, preserving the handful of highly important weights becomes increasingly determinant for model performance.

Quantization location visualizations further reveal that both static (SEPTQ) and dynamic (as in GPTQ) strategies yield similar global patterns; however, the static approach is substantially more computationally efficient.

Figure 2: Comparison of static (SEPTQ) and dynamic (GPTQ) quantization location distributions; both methods converge to similar preserved weight clusters.

Efficient Error-Protected Quantization and Update

SEPTQ employs a column-wise quantization process. For each column, quantized weights are updated per mask; for preserved weights, full precision is retained. Quantization errors are optimally redistributed across the unquantized segments of each row using the analytical solution of a corresponding constrained minimization problem. This error compensation mechanism, informed by the weight importance mask, effectively curtails degradation from aggressive quantization.

Figure 3: Schematic visualization of the quantization and compensation process: preserved weights remain unaltered, quantized weights are updated, and subsequent elements are compensated to protect model outputs.

Empirical Validation and Numerical Outcomes

SEPTQ is validated on OPT, LLaMA, and LLaMA2 model series, across C4, WikiText2, PTB, and a suite of zero-shot downstream benchmarks. The paradigm consistently surpasses strong baselines (GPTQ, QuIP, SpQR, AWQ, AQLM, OmniQuant), with particularly notable performance in the ultra-low (2-bit) quantization regime.

Key results:

• For OPT-66B at 2-bit quantization on C4, WikiText2, and PTB, SEPTQ achieves perplexities of 11.96, 9.93, and 14.93, respectively, compared to GPTQ's 253.49, 619.32, and 630.42, and QuIP's 17.64, 13.86, and 17.50. SEPTQ at 2-bit approaches full-precision performance by merely reserving 0.1% of weights.
• On LLaMA and LLaMA2 zero-shot tasks, SEPTQ at 2(+) bits delivers a 4–28% absolute accuracy increase over GPTQ and 4–8% over QuIP at comparable compression rates, making it viable for robust LLM deployment under severe memory constraints.
• When reserving just 0.1% of parameters (2.01-bit effective), SEPTQ consistently maintains or exceeds the accuracy of full 2-bit and sometimes even 3-bit baselines.

  Figure 4: WikiText2 perplexity on LLaMA-13B for multiple PTQ methods; note SEPTQ's substantially lower degradation at 2-bit.

  

  Figure 5: Perplexity on C4 and WikiText2 for OPT-13B, LLaMA-13B, and LLaMA2-13B; SEPTQ (2.01-bit) outperforms 2-bit GPTQ and QuIP.

Efficiency and Ablation

SEPTQ's static selection of quantization locations leads to notable runtime improvements over dynamic techniques, especially as model size scales. Experimental ablation indicates that both static versus dynamic and global versus local selection strategies impact both speed and perplexity; static/global strategies consistently outperform alternatives in efficiency and accuracy.

Figure 6: Runtime comparison between static and dynamic quantization strategies; static achieves substantial speedup for large models.

Practical and Theoretical Implications

SEPTQ sets a new standard for low-bit PTQ of LLMs, demonstrating that careful weight importance analysis and static masking permit extreme compression without catastrophic accuracy loss. Practically, this enables the deployment of billion-parameter LLMs on single-GPU or edge-class hardware with minimal degradation, opening new application avenues in edge computing and privacy-preserving on-device inference.

Theoretically, the identification of a heavy-tailed distribution in quantization sensitivity underscores deep redundancy in over-parameterized models, and the efficacy of global static selection calls into question the necessity of complex iterative or block-local masking protocols.

Future Directions

SEPTQ suggests that further refinements to static global importance estimation, combined with more sophisticated mask parameterizations, could further reduce the quantization baseline, approaching the theoretical minimal irreducible bit-width for a given accuracy tolerance. Investigating alignment with specialized quantization grid constructions, adaptive mask sparsity, and co-optimization with quantized activation functions are promising directions. Moreover, holistic integration with quantization-aware regularizers during upstream pretraining may yield synergistic effects.

Conclusion

SEPTQ provides an efficient and accurate PTQ paradigm for LLMs, enabling sub-3-bit compression with negligible performance loss. Through its static, global importance-guided masking and optimal error compensation, it establishes strong empirical and methodological baselines that will inform both theoretical understanding and practical deployment of quantized LLMs.