---
title: Activation Decomposition via QUAD
url: https://www.emergentmind.com/topics/activation-decomposition-via-quad
type: topic
---

# Activation Decomposition via QUAD

Activation Decomposition via QUAD is a quantization framework for large language models (LLMs) that addresses the inherent challenge of activation outliers degrading quantization effectiveness, particularly in medium-scale architectures. QUAD—Quantization with Activation Decomposition—incorporates singular value decomposition (SVD) to orthogonally transform the activation space, enabling the isolation and full-precision retention of energetically-dominant (outlier) dimensions, while the remainder of activations and associated weights are quantized to 4 bits. The methodology also introduces a parameter-efficient fine-tuning paradigm wherein only a small submatrix corresponding to outlier weights is adapted. This approach achieves near-lossless quantization accuracy and efficient model adaptation with minimal resource overhead [2503.19353].

## 1. Background and Motivation

LLMs exhibit a significant challenge for efficient deployment due to their scale-induced computational and memory costs. While post-training quantization provides a pathway to reduce these costs, prevalent strategies—such as uniform 4-bit quantization of both weights and activations (W4A4)—are hampered by the presence of rare yet substantial activation outliers. Standard quantization techniques lead to sharp accuracy degradation in medium-scale models (e.g., Llama-3-8B), manifesting as up to 6% reductions relative to FP16 baselines. These effects are attributed to the inadequate representation of energetic activation modes in low-bit quantization, motivating activation decomposition as a means to decouple and treat outliers distinctly [2503.19353].

## 2. Activation Decomposition via Singular Value Decomposition

Let $A \in \mathbb{R}^{B \times C}$ denote layer activations, where $B$ is batch size multiplied by sequence length, and $C$ is hidden dimension. QUAD applies a thin SVD:
\[
A = U \Sigma V^\top
\]
where $U \in \mathbb{R}^{B \times C}$, $\Sigma = \operatorname{diag}(\sigma_1, \dots, \sigma_C)$, $V \in \mathbb{R}^{C \times C}$, with singular values $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_C$. The top-$r$ right singular vectors $V_r = [v_1, \dots, v_r] \in \mathbb{R}^{C \times r}$ capture principal activation directions. The orthogonal complement is defined as $V_{\perp} = I_C - V_r V_r^\top$. An orthogonal transformation matrix $P \in \mathbb{R}^{C \times (C+r)}$ is constructed:
\[
P = [V_r, V_\perp]
\]
This transform $x \mapsto P^\top x$ isolates outlier information in the first $r$ dimensions and preserves isometry ($P^\top P = I_{C+r},\, PP^\top = I_C$), ensuring no expressivity loss.

## 3. Offline Calibration of Principal Activation Subspace

To extract the top singular vectors required for activation decomposition, QUAD aggregates second-moment activation statistics using a modest calibration set. The methodology performs, for each calibration sample, an accumulation $M \leftarrow M + A^\top A$. SVD is then computed on $M$ to obtain principal singular vectors $V_r$. The procedure is efficiently implemented in a single matrix pass. The exact pseudocode outlined in QUAD specifies:

```
Input: model, calibration_data, layer_index i, rank r
Initialize M ← 0_(C×C)
For each sample x in calibration_data do
  Run model up to layer i to get activation A ∈ ℝ^(B×C)
  M ← M + Aᵀ A
end
[U, Σ, _] ← SVD(M)   # U ∈ ℝ^(C×C), Σ diagonal
V_r ← U[:, 1:r]      # top-r singular vectors
V_⊥ ← I_C - V_r V_rᵀ
P ← [V_r, V_⊥]       # shape C×(C+r)
Return P
```

The resulting $P$ transforms incoming activations at inference.

## 4. Activation and Weight Quantization with Outlier Isolation

During inference, activations are projected as $\widehat{A} = A P \in \mathbb{R}^{B \times (C+r)}$. This is partitioned:
\[
\widehat{A} = [A_{\rm out}; A_q],\quad A_{\rm out} \in \mathbb{R}^{B \times r},\, A_q \in \mathbb{R}^{B \times C}
\]
The outlier dimensions $A_{\rm out}$ are retained in FP16, while the remaining $A_q$ dimensions are quantized to 4-bit integers using per-token symmetric rounding-to-nearest (RTN):
\[
s_j = \max_i |(A_q)_{i,j}|/7,\quad (A_q)_{\text{int4}} = \operatorname{round}(A_q / s)
\]
with $(A_q)_{\text{int4}} \in \{-7, \dots, 7\}$ and $A_q^{\text{recon}} = s \cdot (A_q)_{\text{int4}}$. The corresponding rows of the weight matrix are quantized with GPTQ for the $C$ quantized dimensions; the $r$ outlier rows remain FP16.

## 5. Parameter-Efficient Fine-Tuning of Outlier Weights

Fine-tuning in QUAD is parameter-efficient: only the small full-precision submatrix $W_r \in \mathbb{R}^{r \times N}$, corresponding to outlier directions in each $U$-type layer, plus any learned per-row scales $s$, are updated. All quantized parameters $\theta_{\text{quant}}$ remain frozen. Adaptation minimizes the cross-entropy loss:

\[
L(\theta) = \mathbb{E}_{(x, y) \sim D}~ \ell( \operatorname{Model}_{\text{quant}}(x; \theta_{\text{quant}}, W_r),\, y )
\]

Gradients $\partial L/\partial W_r$ are computed via the straight-through estimator to bypass quantization non-differentiability, followed by standard SGD/Adam updates:
\[
W_r \gets W_r - \eta \frac{\partial L}{\partial W_r},\quad s \gets s - \eta \frac{\partial L}{\partial s}
\]
Proposition 4.3 in the source demonstrates that adapting $W_r = V_r^\top W$ is a sub-optimal but effective adapter for approximating full-model fine-tuning in stationary input distributions [2503.19353].

## 6. Empirical Results and Quantitative Performance

QUAD achieves the following zero-shot average accuracies (relative to full-precision FP16 baseline) on Llama-3-8B and Qwen-2.5-7B:

| Model          | Full FP16     | QUAD W4A4        | QUAD W4A4/A8      | + Parameter-Efficient Tuning |
|----------------|--------------|------------------|-------------------|-----------------------------|
| Llama-3-8B     | 73.30        | 68.76 (93.8%)    | 70.71 (96.5%)     | 72.14 (98.4%)               |
| Qwen-2.5-7B    | 71.95        | 69.00 (95.9%)    | 70.54 (98.0%)     | 72.40 (100.6%)              |

By decomposing activations and isolating outliers, QUAD enables 4-bit quantization for both weights and activations with at most a $6\%$ accuracy loss, recoverable to at least $98\%$ of baseline with efficient adapter-style tuning.

## 7. Theoretical Guarantees and Rationale

The QUAD scheme removes the top-$r$ singular components, which empirically shrinks the Frobenius norm $\|A_q\|_F$, reducing the upper bound on quantization error (by Proposition 4.1 in the source). The orthogonality of $P$ ensures the overall expressivity is preserved, avoiding representational loss. Fine-tuning exclusively $W_r$ focuses adaptation capacity into the most energetically significant directions, thus rendering small updates maximally effective [2503.19353].

Source: https://www.emergentmind.com/topics/activation-decomposition-via-quad