Activation Decomposition via QUAD

• Activation Decomposition via QUAD is a quantization framework that enhances large language model performance by isolating energetic activation outliers using SVD.
• It employs efficient calibration and a single matrix pass to capture principal activation directions for optimal quantization.
• The method enables parameter-efficient fine-tuning by updating only a small full-precision submatrix, achieving near-baseline accuracy.

Activation Decomposition via QUAD is a quantization framework for LLMs that addresses the inherent challenge of activation outliers degrading quantization effectiveness, particularly in medium-scale architectures. QUAD—Quantization with Activation Decomposition—incorporates singular value decomposition (SVD) to orthogonally transform the activation space, enabling the isolation and full-precision retention of energetically-dominant (outlier) dimensions, while the remainder of activations and associated weights are quantized to 4 bits. The methodology also introduces a parameter-efficient fine-tuning paradigm wherein only a small submatrix corresponding to outlier weights is adapted. This approach achieves near-lossless quantization accuracy and efficient model adaptation with minimal resource overhead (Hu et al., 25 Mar 2025).

1. Background and Motivation

LLMs exhibit a significant challenge for efficient deployment due to their scale-induced computational and memory costs. While post-training quantization provides a pathway to reduce these costs, prevalent strategies—such as uniform 4-bit quantization of both weights and activations (W4A4)—are hampered by the presence of rare yet substantial activation outliers. Standard quantization techniques lead to sharp accuracy degradation in medium-scale models (e.g., Llama-3-8B), manifesting as up to 6% reductions relative to FP16 baselines. These effects are attributed to the inadequate representation of energetic activation modes in low-bit quantization, motivating activation decomposition as a means to decouple and treat outliers distinctly (Hu et al., 25 Mar 2025).

2. Activation Decomposition via Singular Value Decomposition

Let $A \in \mathbb{R}^{B \times C}$ denote layer activations, where $B$ is batch size multiplied by sequence length, and $C$ is hidden dimension. QUAD applies a thin SVD: $A = U \Sigma V^\top$ where $U \in \mathbb{R}^{B \times C}$, $$\Sigma = \operatorname{diag}(\sigma_1, \dots, \sigma_C)$$, $V \in \mathbb{R}^{C \times C}$, with singular values $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_C$. The top-$r$ right singular vectors $$V_r = [v_1, \dots, v_r] \in \mathbb{R}^{C \times r}$$ capture principal activation directions. The orthogonal complement is defined as $B$0. An orthogonal transformation matrix $B$1 is constructed: $B$2 This transform $B$3 isolates outlier information in the first $B$4 dimensions and preserves isometry ($B$5), ensuring no expressivity loss.

3. Offline Calibration of Principal Activation Subspace

To extract the top singular vectors required for activation decomposition, QUAD aggregates second-moment activation statistics using a modest calibration set. The methodology performs, for each calibration sample, an accumulation $B$6. SVD is then computed on $B$7 to obtain principal singular vectors $B$8. The procedure is efficiently implemented in a single matrix pass. The exact pseudocode outlined in QUAD specifies:

$U \in \mathbb{R}^{B \times C}$3

The resulting $B$9 transforms incoming activations at inference.

4. Activation and Weight Quantization with Outlier Isolation

During inference, activations are projected as $C$0. This is partitioned: $C$1 The outlier dimensions $C$2 are retained in FP16, while the remaining $C$3 dimensions are quantized to 4-bit integers using per-token symmetric rounding-to-nearest (RTN): $C$4 with $C$5 and $C$6. The corresponding rows of the weight matrix are quantized with GPTQ for the $C$7 quantized dimensions; the $C$8 outlier rows remain FP16.

5. Parameter-Efficient Fine-Tuning of Outlier Weights

Fine-tuning in QUAD is parameter-efficient: only the small full-precision submatrix $C$9, corresponding to outlier directions in each $A = U \Sigma V^\top$0-type layer, plus any learned per-row scales $A = U \Sigma V^\top$1, are updated. All quantized parameters $A = U \Sigma V^\top$2 remain frozen. Adaptation minimizes the cross-entropy loss:

$A = U \Sigma V^\top$3

Gradients $A = U \Sigma V^\top$4 are computed via the straight-through estimator to bypass quantization non-differentiability, followed by standard SGD/Adam updates: $A = U \Sigma V^\top$5 Proposition 4.3 in the source demonstrates that adapting $A = U \Sigma V^\top$6 is a sub-optimal but effective adapter for approximating full-model fine-tuning in stationary input distributions (Hu et al., 25 Mar 2025).

6. Empirical Results and Quantitative Performance

QUAD achieves the following zero-shot average accuracies (relative to full-precision FP16 baseline) on Llama-3-8B and Qwen-2.5-7B:

Model	Full FP16	QUAD W4A4	QUAD W4A4/A8	+ Parameter-Efficient Tuning
Llama-3-8B	73.30	68.76 (93.8%)	70.71 (96.5%)	72.14 (98.4%)
Qwen-2.5-7B	71.95	69.00 (95.9%)	70.54 (98.0%)	72.40 (100.6%)

By decomposing activations and isolating outliers, QUAD enables 4-bit quantization for both weights and activations with at most a $A = U \Sigma V^\top$7 accuracy loss, recoverable to at least $A = U \Sigma V^\top$8 of baseline with efficient adapter-style tuning.

7. Theoretical Guarantees and Rationale

The QUAD scheme removes the top-$A = U \Sigma V^\top$9 singular components, which empirically shrinks the Frobenius norm $U \in \mathbb{R}^{B \times C}$0, reducing the upper bound on quantization error (by Proposition 4.1 in the source). The orthogonality of $U \in \mathbb{R}^{B \times C}$1 ensures the overall expressivity is preserved, avoiding representational loss. Fine-tuning exclusively $U \in \mathbb{R}^{B \times C}$2 focuses adaptation capacity into the most energetically significant directions, thus rendering small updates maximally effective (Hu et al., 25 Mar 2025).