---
title: Twin-Log Quantization (TLQ)
url: https://www.emergentmind.com/topics/twin-log-quantization-tlq
type: topic
---

# Twin-Log Quantization (TLQ)

Twin-Log Quantization (TLQ) is a log-based post-training quantization method introduced within LRQ-DiT for low-bit compression of Diffusion Transformers (DiTs) used in text-to-image generation. It is designed for the weight path of linear layers in attention and feed-forward blocks, and addresses a specific failure mode of extreme low-bit PTQ: model weights exhibit a Gaussian-like distribution with long tails, so uniform quantization allocates intervals poorly and induces large reconstruction error. TLQ maps weight magnitudes into the base-2 log domain, splits positive and negative values into separate branches, quantizes each branch into an unsigned $(b-1)$-bit range, and reconstructs weights by exponentiation. In LRQ-DiT, TLQ is paired with an Adaptive Rotation Scheme (ARS) for activations, so that weights and activations are treated by complementary mechanisms rather than by a single quantizer [2508.03485].

## 1. Problem setting and design objective

Diffusion Transformers have achieved strong performance in text-to-image generation, but their computational cost and parameter size create difficulties in resource-constrained scenarios. Post-training quantization is a natural response to these constraints, yet existing PTQ methods are reported to suffer severe degradation under extreme low-bit settings. LRQ-DiT identifies two obstacles: first, model weights follow a Gaussian-like distribution with long tails; second, activations contain two kinds of outliers, namely Mild Outliers with slightly elevated values and Salient Outliers with large magnitudes concentrated in specific channels [2508.03485].

TLQ is the component directed at the first obstacle. Its role is not to quantize the full model state indiscriminately, but specifically to make the weight representation better aligned with the empirical distribution of DiT weights. The core premise is that a log-domain treatment of magnitudes, combined with sign separation and clipping, reduces the mismatch between quantization intervals and the density of values. This positioning is central to LRQ-DiT: TLQ handles weights, while ARS handles activations.

## 2. Mathematical construction

Let $W\in\mathbb{R}^{m\times n}$ be a weight matrix with both positive and negative entries. TLQ first applies a base-2 logarithm to the absolute value,
\[
W'=\log_2\bigl(\lvert W\rvert\bigr).
\]
It then defines binary masks
\[
M^{+}_{ij}=\begin{cases}1&\text{if }W_{ij}>0,\\0&\text{otherwise},\end{cases}
\qquad
M^{-}_{ij}=\begin{cases}1&\text{if }W_{ij}<0,\\0&\text{otherwise},\end{cases}
\]
and forms the sign-split log-magnitudes
\[
W^{+}=W'\odot M^{+},
\qquad
W^{-}=W'\odot M^{-}.
\]

Given a target bit-width $b$, TLQ reserves one bit per sign, so each of $W^{+}$ and $W^{-}$ is quantized into an unsigned $(b-1)$-bit range. With scale factors $s^{+},s^{-}$ and zero-points $z^{+},z^{-}$, Equation (4) is
\[
\begin{aligned}
W_q^{+}
&=\mathrm{clamp}\Bigl(\lfloor W^{+}/s^{+}\rceil-z^{+},\;0,\;2^{\,b-1}-1\Bigr),\\
W_q^{-}
&=\mathrm{clamp}\Bigl(\lfloor W^{-}/s^{-}\rceil-z^{-},\;0,\;2^{\,b-1}-1\Bigr),\\
W_q
&=W_q^{+}-W_q^{-}.
\end{aligned}
\]
This construction preserves sign information by quantizing the positive and negative branches separately rather than pooling all magnitudes into one codebook.

After quantization, floating-point reconstruction is performed by exponentiating the quantized log-magnitudes. Equation (6) is
\[
W_f
=
2^{\,s^{+}(W_q^{+}+z^{+})}\odot M^{+}
-
2^{\,s^{-}(W_q^{-}+z^{-})}\odot M^{-}.
\]
The reconstructed tensor is therefore obtained by inverse mapping from log space, with the original sign pattern reimposed through the masks [2508.03485].

## 3. Clipping, calibration, and algorithmic workflow

TLQ includes a search-based clipping stage intended to suppress extreme tail values. Two clipping hyperparameters, $\alpha,\beta\in(0,1]$, are introduced and chosen by a short grid search to minimize reconstruction error. The optimization target is
\[
\lVert W-W_f(\alpha,\beta)\rVert_2.
\]
The best $\alpha,\beta$ are then used to compute the scale factors and zero-points in Equation (5).

The PTQ workflow is specified for a pre-trained DiT model with a small calibration set of 4–10 prompts. For each weight tensor $W$, the procedure is: compute $W'=\log_2|W|$ and the masks $M^+,M^-$; search over a small grid of $(\alpha,\beta)$ clipping factors to minimize $\lVert W-W_f(\alpha,\beta)\rVert_2$; compute $(s^+,z^+)$ and $(s^-,z^-)$ via Equation (5); quantize the log-magnitudes to integers $W_q^+,W_q^-$ via Equation (4); and store the pair $(f^+,\mathbb I^{r+})$, $(f^-,\mathbb I^{r-})$ for each entry according to Equation (8). At inference time, activations $X$ are quantized separately, and the integer multiply $\mathbb I^X\cdot[f^\pm,\mathbb I^{r\pm}]$ is carried out via fast SHIFTs and AND-masks per Equation (9) [2508.03485].

The integer-friendly path is motivated by hardware cost. Instead of evaluating exponentials directly, TLQ decomposes the exponent term into integer and fractional parts and approximates the fractional power with a small-integer representation. The paper gives the example $I=6$ for $2^{-6}=1/64$. This makes the reconstruction-compatible multiply implementable through bit-shift-based integer arithmetic rather than costly hardware exponentials.

## 4. Distributional rationale and error properties

The theoretical motivation for TLQ is framed against uniform quantization of a zero-mean, long-tailed Gaussian weight distribution $W\sim\mathcal N(0,\sigma^2)$. Uniform quantization uses equal-width intervals across $[W_{\min},W_{\max}]$, but most weights lie near zero. As a result, only a small fraction of the intervals are allocated where the density is largest, producing high $L_2$ distortion [2508.03485].

TLQ changes this geometry by moving to the log domain. The paper states that in log-space the PDF
\[
\rho_{W'}(u)\propto 2^{u}\exp(-2^{2u}/(2\sigma^2))
\]
has a sharper peak and lighter tails. Uniform quantization in $u$ therefore allocates more intervals around small $\lvert W\rvert$ and fewer at extremes. After inverse mapping, intervals become denser near zero, matching the Gaussian peak in weights, while long tails are de-emphasized both by the logarithm and by clipping.

A single log quantizer might appear sufficient, but the paper explicitly rejects that option because pooling positive and negative magnitudes would lose sign information. The “twin-log” construction resolves that issue by splitting sign after the log transform. This is a structural distinction rather than a minor implementation choice: sign preservation and density matching are achieved simultaneously.

The reported error reduction is substantial under extreme low-bit settings. For PixArt weights under 3-bit quantization, the paper states that
\[
\lVert W_{\text{TLQ},3\text{-bit}}-W\rVert_2
\ll
\lVert W_{\text{uniform},3\text{-bit}}-W\rVert_2,
\]
with roughly 3× lower error, and connects this reduction to dramatically better downstream FID scores at 3 bits [2508.03485].

## 5. Function within LRQ-DiT

LRQ-DiT is organized around two complementary PTQ components: Twin-Log Quantization for all linear weights in attention and feed-forward layers, and Adaptive Rotation Scheme for activations. During calibration, TLQ parameters $(\alpha,\beta,s^\pm,z^\pm)$ are computed once per weight tensor. In each forward pass, incoming activations are quantized to $\mathbb I^X$ at 8-bit or lower using channel- or token-wise scales, after possibly applying a rotation selected by ARS; the TLQ-quantized weights are then used through the integer mechanism of Equation (9) [2508.03485].

ARS is driven by a fluctuation metric $J(X)$ and selects between two rotation regimes. When $J$ is small, it applies a simple Hadamard rotation $X\mapsto XH$, which the paper associates with taming Mild outliers. When $J$ exceeds a threshold, ARS switches to DuQuant-style permutation $P$ together with greedy channel rotations $\hat R_1,\hat R_2$, which are used to tame Salient outliers. The division of labor is explicit: TLQ addresses the weight-distribution problem, whereas ARS addresses activation outliers.

This separation also clarifies what TLQ is not. It is not presented as a stand-alone cure for activation quantization pathologies, and LRQ-DiT does not attribute activation robustness to log-domain weight quantization alone. The framework claims synergy instead: clean weights allow sharper activation quantization, and vice versa. A plausible implication is that the reported gains depend on this coordinated treatment of both operands, especially in the lowest-bit regimes.

## 6. Empirical performance and interpretive boundaries

Across PixArt-Σ, PixArt-α, and FLUX models, TLQ+ARS is reported to outperform prior PTQ baselines including SmoothQuant, QuaRot, and ViDiT-Q, especially under 3-bit weight settings. On PixArt-Σ with 20 steps on COCO in the W3A4 setting, the paper gives the following progression: Uniform / no ARS yields FID≈437 and IR≈–2.28; QuaRot only yields FID≈275 and IR≈–2.03; ViDiT-Q only yields FID≈211 and IR≈–1.33; and TLQ+ARS yields FID≈90.15, IR≈+0.43, SSIM≈0.52, and PSNR≈13.47 [2508.03485].

The same source reports that at W3A6, W3A8, and in mixed W4A4–W4A8 regimes, TLQ+ARS preserves high fidelity with FID < 60, IR > 0.8, SSIM > 0.53, and PSNR > 13.3, whereas other methods collapse or degrade sharply. Similar gains are reported on MJHQ and sDCI, and on FLUX 1-schnell/dev at both 4 and 50 steps.

The ablation evidence is presented in compact form. Table 4 is summarized as showing that ARS alone improves FID marginally, TLQ alone cuts FID by ≈2×, and TLQ+ARS cuts FID by ≈4× versus no-quant. In visual comparisons, TLQ+ARS is said to restore crisp detail, accurate colors, and sharp edges under extreme low-bit quantization, making 3-bit DiT inference practical on edge devices.

Two recurring misconceptions are directly addressed by the paper’s construction. First, TLQ should not be conflated with activation quantization; activation robustness in LRQ-DiT is attributed to ARS. Second, TLQ is not merely “log quantization with a sign bit”: its defining feature is the sign-split quantization of log-magnitudes into two unsigned branches, together with clipping and integer-friendly reconstruction. Within the evidence reported for PixArt and FLUX on COCO, MJHQ, and sDCI, this design is presented as the mechanism that allows low-bit DiT quantization while preserving image quality [2508.03485].

Source: https://www.emergentmind.com/topics/twin-log-quantization-tlq