--- title: 'SignRoundV2: Low-Bit LLM Quantization' url: https://www.emergentmind.com/topics/signroundv2 type: topic --- # SignRoundV2: Low-Bit LLM Quantization SignRoundV2 is a post-training quantization (PTQ) framework designed to enable extremely low-bit quantization (2–5 bits) of large language models (LLMs) with minimal loss in accuracy relative to full-precision baselines. It introduces two principal innovations: a first-order, gradient-informed layerwise sensitivity metric (“DeltaLoss”) that directs bit allocation, and a lightweight, calibration-based search for optimal scale initialization prior to quantization. SignRoundV2 demonstrates production-grade performance—within ~1% of full-precision—in the 4–5 bit regime and delivers strong results even with 2-bit weight quantization, advancing the state of efficient LLM deployment [2512.04746]. ## 1. Mathematical Formulation SignRoundV2 employs a symmetric, scale-driven quantizer for both weights and activations. For a full-precision tensor $x$, quantization bit-width $b$, and scale $s$, the quantize-dequantize operator is defined as: $$ qdq(x; b, s) = s \cdot \mathrm{clip} \left( \left\lfloor \frac{x}{s} \right\rceil; n, m \right) $$ where $\lfloor \cdot \rceil$ is round-to-nearest; $\mathrm{clip}(\cdot; n, m)$ saturates to $[n, m]$ with $n = -2^{b-1}$, $m = 2^{b-1}-1$; and $s = \frac{\max(x) - \min(x)}{2^{b-1}}$ (Equation 2). SignRoundV2 maintains compatibility with the trainable offsets ($v, \alpha, \beta$) introduced in SignRound V1 while focusing on initialization and bit allocation. For each weight tensor $W$, the quantization deviation is defined: $$ \Delta_Q(W) = W_f - qdq(W) $$ The sensitivity of each layer $\ell$ is estimated via a first-order Taylor expansion as: $$ \Delta L_\ell \approx \langle g_{aq}, A_f - A_q \rangle + \langle g_{wq}, W_f - W_q \rangle $$ where $g_{aq} = \partial L / \partial A_q$, $g_{wq} = \partial L / \partial W_q$, $A_f$, $A_q$ are full-precision and quantized activations, $W_f$, $W_q$ are full-precision and quantized weights (Equation 3). In practice, activation errors dominate, leading to the DeltaLoss sensitivity metric: $$ S_\ell = \Vert | g_{aq} \circ (A_f - A_q) | \Vert_1 $$ as in Equation 4. ## 2. Algorithmic Components ### 2.1 Layer-wise Bit Allocation SignRoundV2 formulates a 0–1 integer program to minimize aggregate layerwise DeltaLoss under an average bit-width constraint. Given $n$ layers, allowed bit-widths $B = \{b_1, ..., b_K\}$, and a global average-bit target $T$, the optimization is: $$ \min \sum_{i=1}^n \sum_{b\in B} \Delta L_i(b) \cdot I_{i,b} $$ subject to $$ \sum_{b} I_{i,b} = 1 \hspace{0.6cm} \forall i; \quad \sum_{i,b} b \cdot I_{i,b} \cdot P_i \leq T \cdot \sum_i P_i; \quad I_{i,b} \in \{0,1\} $$ where $P_i$ is the parameter count in layer $i$. This is efficiently solved using dynamic programming in $O(n \cdot |B| \cdot \text{budget})$ time. | Step | Description | Time Complexity | |-------------------------|----------------------------------------------------------------|--------------------------| | DeltaLoss computation | Compute $\Delta L_i(b)$ for all $i, b$ | $\mathcal{O}(n|B|)$ | | DP bit allocation | Optimize per-layer bits under global constraint | $\mathcal{O}(n|B|\text{budget})$| ### 2.2 Lightweight Pre-Tuning Scale Search A calibration-driven, grid-based search finds an effective scale $s$ for each layer before any gradient-based tuning. The initialization objective is: $$ \mathrm{Loss}_{init}(s) = \frac{1}{N} \sum_{i=1}^N \left\Vert \left( W_f[i] - qdq(W_f[i]; b, s) \right) \circ \bar{A}^2 \right\Vert_2^2 $$ (Equation 6), where $\bar{A}$ collects per-input-channel activation maxima from a small calibration set. Candidate scales $S$ are scanned, and the minimizer is chosen as $s_{init}$. This is optionally refined by a trainable scalar $\alpha\in [0.5, 1.5]$ for fine-tuning. ### 2.3 Tuning Procedure Each transformer block undergoes 200 sign-gradient descent steps (or up to 500 in extended “Ours*” experiments) on a blockwise reconstruction MSE loss. The learning rate is $1/\text{steps}$, with batch size 8 and sequence length 2048, using mixed precision for improved computational throughput. To reduce outlier influence, the largest $k=0.1\%$ of squared errors in each block are excluded. ## 3. Evaluation: Models, Benchmarks, and Results ### 3.1 Models and Tasks SignRoundV2 has been evaluated on LLaMA 2 (7B, 13B, 70B), LLaMA 3 (8B, 70B), and Qwen (2.5B, 8B, 32B) models. The benchmark suite includes ARC-Easy, ARC-Challenge, BoolQ, HellaSwag, LAMBADA, MMLU, OpenBookQA, PIQA, TruthfulQA, and WinoGrande. ### 3.2 Quantitative Performance For 2-bit weights (W2A16): | Method | LLaMA2-7B | LLaMA2-13B | LLaMA2-70B | |----------------|-----------|------------|------------| | GPTQ (W2A16) | 41.6% | 48.3% | 34.4% | | AWQ (W2A16) | 34.7% | 36.0% | 35.5% | | OmniQ (W2A16) | 47.0% | 53.6% | 54.9% | | SignRound V1 | 54.5% | 60.7% | 67.7% | | SignRound V2 | 57.9% | 61.9% | 68.4% | At 4–5 bits average (MXFP4/8), SignRoundV2 achieves 95–99% recovery of full-precision accuracy, sustaining $\sim1\%$ variance. ## 4. Ablation Studies and Comparative Analysis ### 4.1 Initialization Initialization with scale pre-tuning yields gains of 5–10 percentage points in average accuracy over “without init” baselines: Qwen3-8B improves from 48–54% to 56–66%; LLaMA3.1-8B from 48–53% to 50–60%. ### 4.2 DeltaLoss-Only vs. Full Tuning The DeltaLoss-only mode (no gradient-based SignRound tuning) already surpasses heuristic methods such as “head-8bit,” “tail-8bit,” and RTN. Full SignRoundV2 further improves accuracy by approximately 1–2 percentage points due to sign-gradient rounding. ### 4.3 Mixed vs. Uniform Precision In pure 2-bit (W2A16) mode, uniform-precision SignRoundV2 nearly closes the gap with mixed-precision setups. For 4–5 bits, uniform allocation plus SignRoundV2 achieves $>95\%$ recovery; mixed-precision yields only marginal gains. ## 5. Implementation and Practical Considerations ### 5.1 Pipeline and Resource Profile SignRoundV2 is available as open source at https://github.com/intel/auto-round, offering routines for DeltaLoss computation, dynamic bit allocation, pre-tuning, and blockwise SignRound training. Default hyperparameters include 200 steps per block, batch size 8, and 128 calibration samples. The pipeline, for one LLM instance, typically proceeds as follows: 1. Load full-precision (FP) model. 2. Collect activation maxima $\bar{A}$ from 16–64 random calibration prompts. 3. Compute DeltaLoss sensitivities ($\sim$5–10 min per 8B model). 4. Solve for per-layer bit assignment. 5. Pre-tune quantization scales by grid search on Eq. 6. 6. Run per-block SignRound tuning ($\sim$2–3 h per 70B model). 7. Export quantized model for inference. ### 5.2 Deployment and Performance - **Weight memory**: W2A16 mode yields 8× reduction; MXFP4/8 mode achieves 4–2× reduction. - **Inference speed**: Quantized matmul kernels (ADSO, standard INT) provide near 2–4× latency improvements on GPU/CPU. - **Resource constraints**: 70B models fit in $\sim 40$GB VRAM (W2A16) with $\sim 10$GB peak overhead for DeltaLoss. A plausible implication is that SignRoundV2 enables practical PTQ of large-scale LLMs on commodity hardware by combining first-order accuracy maintenance and computational efficiency. ## 6. Significance and Future Directions SignRoundV2 establishes two major contributions for low-bit LLM quantization: (1) a scalable, gradient-informed sensitivity metric (DeltaLoss) that guides allocation, and (2) efficient pre-tuning that substantially improves scale initialization, both of which lead to robust quantization even in extremely low-bit regimes. Its methodology generalizes to a range of LLM architectures and could be further extended by investigation into alternative sensitivity metrics, broader calibration strategies, or adaptation to even lower resource targets [2512.04746]. Source: https://www.emergentmind.com/topics/signroundv2