---
title: 'LRQ-DiT: Low-Bit PTQ for Diffusion Transformers'
url: https://www.emergentmind.com/topics/lrq-dit
type: topic
---

# LRQ-DiT: Low-Bit PTQ for Diffusion Transformers

Searching arXiv for the cited LRQ-DiT paper and closely related DiT PTQ work to ground the article.
LRQ-DiT denotes **“Log-Rotation Post-Training Quantization of Diffusion Transformers for Text-to-Image Generation”**, a post-training quantization (PTQ) framework for **Diffusion Transformers (DiTs)** that targets resource-constrained deployment of text-to-image models such as **PixArt** and **FLUX** [2508.03485]. The method is explicitly **training-free** and **calibration-based**, and it is designed for **extreme low-bit settings** where existing PTQ methods for DiTs suffer severe quality degradation. Its central premise is that DiT quantization is limited by two structural obstacles: **Gaussian-like long-tailed weight distributions** and **two activation outlier types**, termed **Mild Outliers** and **Salient Outliers**. LRQ-DiT addresses these with two coordinated modules: **Twin-Log Quantization (TLQ)** for weights and an **Adaptive Rotation Scheme (ARS)** for activations [2508.03485].

## 1. Nomenclature and scope

LRQ-DiT is a **DiT quantization** method rather than a solver, accelerator, or training system. The acronym can be confused with several unrelated methods that share either the “LRQ” or “DiT” naming pattern.

The most direct source of ambiguity is **LRQ-Solver**, whose full name is **“Low-Rank Query-based PDE Solver”**. That model is a **transformer-based neural operator** for large-scale 3D PDEs, built around **Parameter-Conditioned Lagrangian Modeling (PCLM)** and **Low-Rank Query Attention (LR-QA)**, and it is described explicitly as *not* being a diffusion model or a DiT-style generative architecture [2510.11636]. A second possible confusion is **$\Delta$-DiT**, which is a **training-free acceleration method tailored for diffusion transformers** via stage-adaptive caching and **$\Delta$-Cache**, rather than a quantization framework [2406.01125]. A third related but distinct method is **Q-DiT4SR**, which is a PTQ framework for **DiT-based real-world image super-resolution**, with emphasis on preserving local textures and timestep-aware activation precision rather than text-to-image generation [2602.01273]. A fourth is **DiT-HC**, a systems paper on efficient **DiT training on HPC-oriented CPU clusters**, centered on communication-free tensor parallelism, optimized kernels, and a custom MPI backend rather than compression [2601.01500].

Within this landscape, LRQ-DiT is specifically a **low-bit PTQ framework for text-to-image DiTs**. Its contribution is not diffusion acceleration, neural PDE solving, or hardware-aware large-scale training, but **post-training compression with minimal calibration and no retraining**.

## 2. Quantization problem formulation and identified failure modes

The problem addressed by LRQ-DiT is the degradation of image quality when DiTs are quantized to **very low bit-widths**. The paper states that existing PTQ methods are workable at moderate settings such as **W4A8** or **W8A8**, but quality collapses in more aggressive regimes such as **3-bit weights** and low-bit activations [2508.03485].

For weights, LRQ-DiT attributes this failure to the fact that DiT parameters are **approximately zero-mean Gaussian-like with long tails**. Under uniform quantization, equal-width bins are allocated across the full numeric range. The paper argues that this is a poor match for DiT weights because the dense center of the distribution receives insufficient resolution, while sparse but large tail values incur large absolute quantization error. The mismatch becomes especially harmful at very low precision.

For activations, the paper identifies **two outlier regimes**. **Mild outliers** are values that slightly exceed typical activation magnitudes and *rarely surpass 5*. **Salient outliers** are much larger values concentrated in **specific channels**. The paper gives a concrete example from one PixArt layer in which **3.6% of values exceed 10**, **0.1% exceed 100**, and the **peak reaches 245** [2508.03485]. Unlike LLM outliers that are often token-specific, these DiT outliers are described as **distributed across tokens** but **concentrated in specific channels**. The distinction matters because it motivates different mitigation strategies for different activation regimes.

This diagnosis structures the entire method. **TLQ** is introduced to better match the empirical shape of the weight distribution, while **ARS** is introduced to suppress activation outliers without uniformly applying expensive transformations to every layer.

## 3. Twin-Log Quantization for weights

**Twin-Log Quantization (TLQ)** is the weight-quantization component of LRQ-DiT. Its purpose is to quantize DiT weights in a representation better aligned with the paper’s observed **Gaussian-like long-tailed** distribution [2508.03485].

The starting point is the paper’s standard affine quantization form,
\[
\mathbf{x}_q = \operatorname{clamp}\left(\left\lfloor \frac{\mathbf{x}}{s} \right\rceil - z,\; 0,\; 2^b-1\right),
\qquad
\mathbf{x}_f = s \cdot (\mathbf{x}_q + z) \approx \mathbf{x},
\]
with
\[
s = \frac{x_{\max}-x_{\min}}{2^b-1},
\qquad
z = \left\lfloor \frac{x_{\min}}{s} \right\rceil.
\]

TLQ replaces direct linear quantization by operating in **log space** and by treating positive and negative weights separately. The paper defines
\[
\mathbf{W}' = \log_2(|\mathbf{W}|),
\]
then splits transformed weights using sign masks,
\[
\mathbf{W}^{+} = \mathbf{W}' \cdot \mathbf{M}^{+},
\qquad
\mathbf{W}^{-} = \mathbf{W}' \cdot \mathbf{M}^{-}.
\]

The positive and negative parts are quantized independently,
\[
\mathbf{W}_{q}^{+} = \operatorname{clamp}\left(\left\lfloor \frac{\mathbf{W}^{+}}{s^{+}} \right\rceil - z^{+},\; 0,\; 2^{b-1}-1\right),
\]
\[
\mathbf{W}_{q}^{-} = \operatorname{clamp}\left(\left\lfloor \frac{\mathbf{W}^{-}}{s^{-}} \right\rceil - z^{-},\; 0,\; 2^{b-1}-1\right),
\]
and reconstructed as
\[
\mathbf{W}_{q} = \mathbf{W}_{q}^{+} - \mathbf{W}_{q}^{-}.
\]

The scales and zero-points are clipping-aware:
\[
s^{+} = \frac{\alpha \cdot \mathbf{W}_{\max}^{+} - \mathbf{W}_{\min}^{+}}{2^{b-1}-1},
\qquad
z^{+} = \left\lfloor \frac{\mathbf{W}_{\min}^{+}}{s^{+}} \right\rceil,
\]
\[
s^{-} = \frac{\beta \cdot \mathbf{W}_{\max}^{-} - \mathbf{W}_{\min}^{-}}{2^{b-1}},
\qquad
z^{-} = \left\lfloor \frac{\mathbf{W}_{\min}^{-}}{s^{-}} \right\rceil,
\]
where \(\alpha\) and \(\beta\) are clipping hyperparameters selected by a **simple grid search**.

The dequantization is also defined in the log domain:
\[
\mathbf{W}_{f}
=
2^{\,s^{+}(\mathbf{W}_{q}^{+}+z^{+})}\cdot \mathbf{M}^{+}
-
2^{\,s^{-}(\mathbf{W}_{q}^{-}+z^{-})}\cdot \mathbf{M}^{-}.
\]

The conceptual role of TLQ is straightforward. Working in log space gives greater effective representational emphasis to small-magnitude values, compresses large values, and allocates finer effective resolution to the dense center of the distribution. The clipping terms \(\alpha\) and \(\beta\) further suppress tail-dominated error. The paper states that this yields a substantially better fit than uniform quantization for low-bit DiT weights [2508.03485].

The paper also includes a **hardware-oriented implementation**. It decomposes the exponents into integer and residual parts,
\[
f^{+} = \left\lfloor s^{+}(\mathbf{W}_{q}^{+}+z^{+}) \right\rfloor,
\qquad
r^{+} = s^{+}(\mathbf{W}_{q}^{+}+z^{+}) - f^{+},
\]
\[
f^{-} = \left\lfloor s^{-}(\mathbf{W}_{q}^{-}+z^{-}) \right\rfloor,
\qquad
r^{-} = s^{-}(\mathbf{W}_{q}^{-}+z^{-}) - f^{-},
\]
and approximates residual powers using an integerization factor \(2^{-I}\) such as \(1/64\), \(1/128\), or \(1/256\). The stated purpose is to make exponentiation amenable to shift/add/bitwise execution [2508.03485].

## 4. Adaptive Rotation Scheme for activations

The activation component of LRQ-DiT is the **Adaptive Rotation Scheme (ARS)**, which is designed to address both **mild** and **salient** outliers [2508.03485]. The method does not apply a single fixed rotation to all layers. Instead, it selects between a lightweight and a stronger transformation according to an activation fluctuation metric.

The metric is
\[
\mathbf{J} = \frac{\|\mathbf{X}\|_F}{\sqrt{BNC}},
\]
where \(\|\mathbf{X}\|_F\) is the Frobenius norm and \(B\), \(N\), and \(C\) are the batch, token/sequence, and channel dimensions. The paper uses this quantity to determine whether a layer exhibits sufficiently strong fluctuation to require outlier-aware treatment.

If \(\mathbf{J} < \text{Threshold}\), ARS uses **Hadamard rotation**,
\[
\mathbf{X} \leftarrow \mathbf{X}\mathbf{R}_H,
\]
where \(\mathbf{R}_H \in \{+1,-1\}^{n \times n}\) is a Hadamard matrix. This path is lightweight and is intended for layers dominated by mild outliers.

If \(\mathbf{J} \ge \text{Threshold}\), ARS switches to a stronger **outlier-aware dual transformation**,
\[
\mathbf{X} \leftarrow \mathbf{X}\hat{\mathbf{R}}_1 \mathbf{P} \hat{\mathbf{R}}_2,
\]
where \(\hat{\mathbf{R}}_1\) is a greedy outlier-aware rotation, \(\mathbf{P}\) is a permutation matrix, and \(\hat{\mathbf{R}}_2\) is a second rotation after permutation. The paper states that this path is adapted from DuQuant-style ideas and is intended to suppress salient channel outliers.

The greedy construction of \(\hat{\mathbf{R}}_1\) begins by identifying the channel with the largest outlier concentration, \(c^*\), and then building an approximate rotation using
\[
\mathbf{R}^{i} = \mathbf{E}_{c^*}\,\tilde{\mathbf{R}}\,\mathbf{Q}\,\mathbf{E}_{c^*},
\qquad
\hat{\mathbf{R}} = \prod_{i=1}^{n}\mathbf{R}^{i},
\]
with \(\mathbf{E}_{c^*}\) a switch matrix, \(\tilde{\mathbf{R}}\) an initialized orthogonal matrix, and \(\mathbf{Q}\) another orthogonal component. The procedure is repeated greedily on the **top-\(k\) influential channels**. After the first rotation, channels are sorted by numerical range and then **alternately assigned across blocks**, so that \(\mathbf{P}\) spreads outlier-heavy channels more evenly before the second rotation smooths residual outliers.

The paper reports that setting the threshold to **1** works well in practice, and that only about **5% of layers** require the more expensive outlier-aware treatment [2508.03485]. This suggests that the method’s adaptivity is central to its efficiency: most layers use a cheap Hadamard transform, while only a small subset invokes the stronger channel-aware path.

## 5. Calibration protocol, evaluation setup, and implementation conventions

LRQ-DiT is described as **training-free** and **calibration-based**. During calibration it uses **4–10 calibration prompts** and estimates the **smoothing factor**, the fluctuation metric \(\mathbf{J}\), the rotation matrix \(\mathbf{R}\), and the permutation matrix \(\mathbf{P}\) [2508.03485]. The paper states explicitly that the method requires **no retraining or parameter finetuning**. It also follows **ViDiT-Q-style settings** in which **sensitive small layers are kept in full precision**.

The experimental setup covers four DiT-family generation configurations:

- **PixArt-\(\Sigma\)**, **20 steps**, **CFG 4.5**
- **PixArt-\(\alpha\)**, **20 steps**, **CFG 4.5**
- **FLUX.1-schnell**, **4 steps**, **CFG 0**
- **FLUX.1-dev**, **50 steps**, **CFG 3.5**

Evaluation is performed on **COCO**, **MJHQ-30K**, and **sDCI**, with **1024 prompts** from each dataset. The reported metrics are **FID**, **IR (ImageReward)**, **PSNR**, and **SSIM** [2508.03485].

The baseline methods are **SmoothQuant**, **QuaRot**, and **ViDiT-Q**, rerun under matched settings for fairness. This positioning is important because LRQ-DiT is not presented as a generic quantization recipe, but as a DiT-specific PTQ framework whose main claims concern **extreme low-bit robustness**.

## 6. Empirical results, ablations, and relation to adjacent DiT compression work

The main empirical claim is that LRQ-DiT preserves image quality substantially better than prior PTQ baselines at aggressive bit-widths, especially **W3A4** [2508.03485]. On **PixArt-\(\alpha\)** at **W3A4** and **COCO**, the paper reports **FID 321.59** for Smooth*, **292.01** for QuaRot*, **245.37** for ViDiT-Q*, and **90.36** for LRQ-DiT. On **MJHQ**, the reported values are **232.04** for ViDiT-Q* and **75.51** for LRQ-DiT; on **sDCI**, **216.56** for ViDiT-Q* and **87.06** for LRQ-DiT. The paper states that the pattern is consistent, with LRQ-DiT dramatically reducing FID and improving **IR**, **SSIM**, and **PSNR**.

At less extreme settings, the gains remain but are smaller. For **PixArt-\(\Sigma\)** at **W4A8** on **COCO**, the paper reports **36.96** for ViDiT-Q* and **35.16** for LRQ-DiT; on **MJHQ**, **34.23** for ViDiT-Q* and **32.39** for LRQ-DiT. This matches the paper’s broader observation that improvements are most dramatic at very low bit-widths.

The FLUX results are described as more modest but still consistent. For **FLUX.1-schnell** at **W3A4**, the reported COCO FIDs are **59.07** for ViDiT-Q* and **50.77** for LRQ-DiT; on **MJHQ**, **63.71** versus **54.52**; on **sDCI**, **62.25** versus **56.40**. For **FLUX.1-dev** at **W3A4** on COCO, the paper reports **91.18** for ViDiT-Q* and **58.81** for LRQ-DiT.

The ablation study isolates the contributions of TLQ and ARS. On **PixArt-\(\alpha\)**, **COCO**, **W3A4**, the paper reports **FID 245.37** with neither component, **237.15** with ARS only, **99.50** with TLQ only, and **90.36** with **TLQ + ARS** [2508.03485]. The paper interprets this as evidence that the two modules are complementary: TLQ contributes most of the improvement, while ARS provides additional gains beyond weight quantization alone.

The qualitative results are described as showing **clearer foreground details**, **better motion scene depiction**, **improved character recognition**, and **better overall image fidelity** than **QuaRot** or **ViDiT-Q**. The paper does not emphasize major limitations, but it does imply several practical considerations: the strongest ARS path still introduces overhead, calibration data and threshold selection remain necessary, the hardware-oriented TLQ formulation relies on exponent approximations, and the largest gains occur in the most aggressive low-bit regimes.

In relation to adjacent work, LRQ-DiT occupies a specific point in the DiT efficiency literature. Unlike **$\Delta$-DiT**, which accelerates diffusion transformers by **stage-adaptive caching** and **$\Delta$-Cache** [2406.01125], LRQ-DiT reduces model precision through PTQ. Unlike **Q-DiT4SR**, which is tuned for **real-world image super-resolution** through **H-SVD**, **VaSMP**, and **VaTMP** [2602.01273], LRQ-DiT is formulated for **text-to-image DiTs** and centers its design on **log-domain weight quantization** and **adaptive activation rotation**. The resulting picture is that LRQ-DiT is best understood as a **DiT-specific low-bit PTQ framework** whose main novelty lies in aligning the quantizer with DiT weight statistics and making activation preprocessing conditional on the observed outlier regime.

Source: https://www.emergentmind.com/topics/lrq-dit